Vision correction system and method, light field display and light field shaping layer and alignment therefor

ABSTRACT

Described are various embodiments of a digital display device for use by a user having reduced visual acuity. In one embodiment, the device comprises: a digital display medium comprising an array of pixels and operable to render a pixelated image accordingly; a light field shaping layer defined by an array of light field shaping elements and disposed relative to said digital display so to align each of said light field shaping elements with a corresponding set of said pixels to shape a light field emanating therefrom and thereby at least partially govern a projection thereof from said display medium toward the user; and a hardware processor operable on pixel data for the image such that said processed image is rendered to at least partially compensate for the user&#39;s reduced visual acuity.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/016,238 filed Sep. 9, 2020, which is a continuation-in-part ofPCT/IB2019/051896 filed Mar. 8, 2019, a continuation-in-part ofPCT/IB2019/051893 filed Mar. 8, 2019, and a continuation-in-part ofPCT/IB2019/051871 filed Mar. 8, 2019. PCT/IB2019/051896 is acontinuation-in-part of U.S. patent application Ser. No. 16/259,845filed Jan. 28, 2019. PCT/IB2019/051871 is a continuation-in-part of U.S.patent application Ser. No. 16/259,845 filed Jan. 28, 2019.PCT/IB2019/051893 is a continuation-in-part of U.S. patent applicationSer. No. 16/259,845 filed Jan. 28, 2019. The entire disclosures of eachof which is hereby incorporated by reference.

FIELD OF THE DISCLOSURE

The present disclosure relates to digital displays, and in particular,to a vision correction system and method, light field display and lightfield shaping layer therefor.

BACKGROUND

Individuals routinely wear corrective lenses to accommodate for reducedvision acuity in consuming images and/or information rendered, forexample, on digital displays provided, for example, in day-to-dayelectronic devices such as smartphones, smart watches, electronicreaders, tablets, laptop computers and the like, but also provided aspart of vehicular dashboard displays and entertainment systems, to namea few examples. The use of bifocals or progresses corrective lenses isalso commonplace for individuals suffering from near and farsightedness.

The operating systems of current electronic devices having graphicaldisplays offer certain “Accessibility” features built into the softwareof the device to attempt to provide users with reduced vision theability to read and view content on the electronic device. Specifically,current accessibility options include the ability to invert images,increase the image size, adjust brightness and contrast settings, boldtext, view the device display only in grey, and for those with legalblindness, the use of speech technology. These techniques focus on thelimited ability of software to manipulate display images throughconventional image manipulation, with limited success.

Light field displays using lenslet arrays or parallax barriers have beenproposed for correcting such visual aberrations. For a thorough reviewof Autostereoscopic or light field displays, Halle M. (Halle, M.,“Autostereoscopic displays and computer graphics” ACM SIGGRAPH, 31(2),pp. 58-62, 1997) gives an overview of the various ways to build aglasses-free 3D display, including but not limited to parallax barriers,lenticular sheets, microlens arrays, holograms, and volumetric displays,for example. Moreover, the reader is also directed to another article byMasia et al. (Masia B., Wetzstein G., Didyk P. and Gutierrez, “A surveyon computational displays: Pushing the boundaries of optics, computationand perception”, Computer & Graphics 37 (2013), 1012-1038), which alsoprovides a good review of computational displays, notably light fielddisplays at section 7.2 and vision correcting light field displays atsection 7.4.

A first example of using light field displays to correct visualaberrations has since been proposed by Pamplona et al. (PAMPLONA, V.,OLIVEIRA, M., ALIAGA, D., AND RASKAR, R. 2012. “Tailored displays tocompensate for visual aberrations.” ACM Trans. Graph. (SIGGRAPH) 31.).Unfortunately, conventional light field displays as used by Pamplona etal. are subject to a spatio-angular resolution trade-off; that is, anincreased angular resolution decreases the spatial resolution. Hence,the viewer sees a sharp image but at the expense of a significantlylower resolution than that of the screen. To mitigate this effect, Huanget al. (see, HUANG, F.-C., AND BARSKY, B. 2011. A framework foraberration compensated displays. Tech. Rep. UCB/EECS-2011-162,University of California, Berkeley, December; and HUANG, F.-C., LANMAN,D., BARSKY, B. A., AND RASKAR, R. 2012. Correcting for opticalaberrations using multi layer displays. ACM Trans. Graph. (SiGGRAPHAsia) 31, 6, 185:1-185:12. proposed the use of multilayer displaydesigns together with prefiltering. The combination of prefiltering andthese particular optical setups, however, significantly reduces thecontrast of the resulting image.

Moreover, in U.S. Patent Application Publication No. 2016/0042501 andFu-Chung Huang, Gordon Wetzstein, Brian A. Barsky, and Ramesh Raskar.“Eyeglasses-free Display: Towards Correcting Visual Aberrations withComputational Light Field Displays”. ACM Transaction on Graphics, xx:0,August 2014, the entire contents of each of which are herebyincorporated herein by reference, the combination of viewer-adaptivepre-filtering with off-the-shelf parallax barriers has been proposed toincrease contrast and resolution, at the expense however, of computationtime and power.

Another example includes the display of Wetzstein et al. (Wetzstein, G.et al., “Tensor Displays: Compressive Light Field Synthesis usingMultilayer Displays with Directional Backlighting”,https://web.media.mit.edu/˜gordonw/TensorDisplays/TensorDisplays.pdf)which discloses a glass-free 3D display comprising a stack oftime-multiplexed, light-attenuating layers illuminated by uniform ordirectional backlighting. However, the layered architecture may cause arange of artifacts including Moiré effects, color-channel crosstalk,interreflections, and dimming due to the layered color filter array.Similarly, Agus et al. (AGUS M. et al., “GPU Accelerated Direct VolumeRendering on an Interactive Light Field Display”, EUROGRAPHICS 2008,Volume 27, Number 2, 2008) discloses a GPU accelerated volume raycasting system interactively driving a multi-user light field display.The display, produced by the Holographika company, uses an array ofspecially arranged projectors and a holographic screen to provideglass-free 3D images. However, the display only provides a parallaxeffect in the horizontal orientation as having parallax in both verticaland horizontal orientations would be too computationally intensive.Finally, the FOVI3D company(http://on-demand.gputechconf.com/gtc/2018/presentation/s8461-extreme-multi-view-rendering-for-light-field-displays.pdf)provides light field displays wherein the rendering pipeline is areplacement for OpenGL which transports a section of the 3D geometry forfurther processing within the display itself. This extra processing ispossible because the display is integrated into a bulky table-likedevice. However, the extra space required by the bulky frame andextensive computation power is unavailable when seeking to implementcomputationally approachable vision correction applications on portabledevices, vehicular dashboards, smartphones, smart watches or other suchconsumer products and/or applications.

This background information is provided to reveal information believedby the applicant to be of possible relevance. No admission isnecessarily intended, nor should be construed, that any of the precedinginformation constitutes prior art.

SUMMARY

The following presents a simplified summary of the general inventiveconcept(s) described herein to provide a basic understanding of someaspects of the invention. This summary is not an extensive overview ofthe invention. It is not intended to restrict key or critical elementsof the invention or to delineate the scope of the invention beyond thatwhich is explicitly or implicitly described by the following descriptionand claims.

A need exists for a vision correction system and method, light fielddisplay and optical element array therefor, that overcome some of thedrawbacks of known techniques, or at least, provide a useful alternativethereto. Some aspects of the disclosure provide embodiments of suchsystems, methods, displays and optical element arrays, such as microlensarrays, or again, using subpixel rendering, for example.

In accordance with one aspect, there is provided a digital displaydevice to render an image for viewing by a viewer having reduced visualacuity, the device comprising: a digital display medium comprising anarray of pixels and operable to render a pixelated image accordingly; alight field shaping layer (LFSL) defined by an array of LFSL elementsand disposed relative to said digital display medium so to dispose eachof said LFSL elements over an underlying set of said pixels to shape alight field emanating therefrom and thereby at least partially govern aprojection thereof from said display medium toward the viewer, whereinsaid LFSL is disposed relative to said digital display medium ingeometric misalignment so to geometrically misalign said array of pixelsand said array of LFSL elements; and a hardware processor operable onpixel data for the input image to output adjusted image pixel data to berendered via said digital display medium and projected through saidgeometrically misaligned LFSL in accordance with said geometricmisalignment so to produce a designated image perception adjustment toat least partially address the viewer's reduced visual acuity.

In one embodiment, the hardware processor is operable to: digitally mapthe input image on an adjusted image plane designated to provide theuser with the designated image perception adjustment; associate saidadjusted image pixel data with at least some of said pixel setsaccording to said mapping and said geometric misalignment; and rendersaid adjusted image pixel data via said pixel sets thereby rendering aperceptively adjusted version of the input image when viewed throughsaid LFSL.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to said digital display medium at adesignated minimum viewing distance designated such that saidperceptively adjusted version of the input image is adjusted toaccommodate the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In one embodiment, the hardware processor is operable to account forsaid geometric misalignment based on a stored mapping of each said LFSLelement relative to said array of pixels.

In one embodiment, the array of pixels is angled or rotated relative tosaid array of LFSL elements.

In one embodiment, one of said array of pixels and said array of LFSLelements is defined by a square array whereas the other is defined by ahexagonal array.

In one embodiment, a pitch ratio between said array of pixels and saidarray of LFSL elements is designated as an irrational number.

In one embodiment, a pitch ratio between said array of pixels and saidarray of LFSL elements is designated so to define an array mismatchpattern that repeats on a periodic distance substantially equal to, orgreater than a dimension of said digital display medium.

In one embodiment, the LFSL comprises at least one of a micro lensarray, a parallax barrier, a diffractive light field barrier or acombination thereof.

In one embodiment, the LFSL comprises a microlens array having a focallength, and wherein said microlens array is disposed at a designateddistance from said digital display that is shorter than said focallength so to produce a divergent light field therefrom.

In accordance with another aspect, there is provided acomputer-implemented method, automatically implemented by one or moredigital processors, to automatically adjust user perception of an inputimage to be rendered on a digital display medium via a set of pixelsthereof to at least partially address a viewer's reduced visual acuity,wherein the digital display medium has a light field shaping layer(LFSL) disposed thereon comprising an array of LFSL elements, whereineach of said LFSL elements is disposed over an underlying set of saidpixels to shape a light field emanating therefrom and thereby at leastpartially govern a projection thereof from said display medium towardthe viewer, the method comprising: digitally mapping the input image onan adjusted image plane designated to provide the user with a designatedimage perception adjustment; associating adjusted image pixel data withat least some of said pixel sets according to said mapping, and furtheraccording to a geometric misalignment of said array of LFSL elements andsaid array of pixels; and rendering said adjusted image pixel data viasaid pixel sets thereby rendering a perceptively adjusted version of theinput image when viewed through said LFSL to at least partially addressthe viewer's reduced visual acuity.

In one embodiment, the associating is implemented according to a storedmapping of each LFSL element relative to the array of pixels.

In one embodiment, the array of pixels is angled or rotated relative tothe array of LFSL elements.

In one embodiment, one of the array of pixels and the array of LFSLelements is defined by a square array whereas the other is defined by ahexagonal array.

In one embodiment, a pitch ratio between the array of pixels and thearray of LFSL elements is designated as an irrational number.

In one embodiment, a pitch ratio between the array of pixels and thearray of LFSL elements is designated so to define an array mismatchpattern that repeats on a periodic distance substantially equal to, orgreater than a dimension of the digital display medium.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to the digital display at a designatedminimum viewing distance designated such that said perceptively adjustedversion of the input image is adjusted to accommodate the viewer'sreduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In accordance with another aspect, there is provided a non-transitorycomputer-readable medium having instructions stored thereon, to beautomatically implemented by one or more digital processors, toautomatically adjust user perception of an input image to be rendered bya digital display medium via a set of pixels thereof to at leastpartially address a viewer's reduced visual acuity, wherein the digitaldisplay medium has a light field shaping layer (LFSL) disposed thereoncomprising an array of LFSL elements, wherein each of said LFSL elementsis disposed over an underlying set of said pixels to shape a light fieldemanating therefrom and thereby at least partially govern a projectionthereof from the display medium toward the viewer, the instructionscomprising instructions to: digitally map the input image on an adjustedimage plane designated to provide the user with a designated imageperception adjustment; associate adjusted image pixel data with at leastsome of said pixel sets according to said mapping, and further accordingto a geometric misalignment of the array of LFSL elements and the arrayof pixels; and render said adjusted image pixel data via said pixel setsthereby rendering a perceptively adjusted version of the input imagewhen viewed through said LFSL to at least partially address the viewer'sreduced visual acuity.

In one embodiment, the associating is implemented according to a storedmapping of each LFSL element relative to the array of pixels.

In one embodiment, the array of pixels is angled or rotated relative tothe array of LFSL elements.

In one embodiment, one of the array of pixels and the array of LFSLelements is defined by a square array whereas the other is defined by ahexagonal array.

In one embodiment, a pitch ratio between the array of pixels and thearray of LFSL elements is designated as an irrational number.

In one embodiment, a pitch ratio between the array of pixels and thearray of LFSL elements is designated so to define an array mismatchpattern that repeats on a periodic distance substantially equal to, orgreater than a dimension of the digital display medium.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to the digital display medium at adesignated minimum viewing distance designated such that saidperceptively adjusted version of the input image is adjusted toaccommodate the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In accordance with another aspect, there is provided a digital displaydevice to render an image for viewing by a viewer having reduced visualacuity, the device comprising: a digital display medium comprising anarray of pixels and operable to render a pixelated image accordingly,wherein each of said pixels comprises a combination of sub pixelsgeometrically disposed and associated therewith; a light field shapinglayer (LFSL) defined by an array of LFSL elements and disposed relativeto said digital display medium so to dispose each of said LFSL elementsover an underlying set of said pixels to shape a light field emanatingtherefrom and thereby at least partially govern a projection thereoffrom said digital display medium toward the viewer; and a hardwareprocessor operable on pixel data for the input image to output adjustedimage pixel data to be rendered via said digital display medium throughsaid LFSL in accordance with a stored alignment thereof so to produce adesignated image perception adjustment to at least partially address theviewer's reduced visual acuity, wherein said hardware processor isoperable to calculate a light field component associated with at leastsome of said subpixels to output independently adjusted subpixel datavalues accordingly.

In one embodiment, the hardware processor is operable to: digitally mapthe input image on an adjusted image plane designated to provide theuser with said designated image perception adjustment; associate saidadjusted image pixel data with at least some of said pixel setsaccording to said mapping; and render said adjusted image pixel data viasaid pixel sets thereby rendering a perceptively adjusted version of theinput image when viewed through said LFSL.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to said digital display medium at adesignated minimum viewing distance designated such that saidperceptively adjusted version of the input image is adjusted toaccommodate the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In one embodiment, the adjusted image pixel data comprises an adjustedpixel colour computed so to manifest a corresponding adjusted imagecolour at a corresponding location on said adjusted image plane givensaid stored alignment.

In one embodiment, the adjusted subpixel data values comprise anadjusted image colour channel value.

In one embodiment, each of said subpixels is associated with a subpixellocation on said digital display medium, and wherein said hardwareprocessor is operable to calculate said light field component associatedwith a given subpixel based on said subpixel location thereof.

In one embodiment, each given pixel is associated with a given storedpixel location on said digital display medium, wherein each givensubpixel for said given pixel is disposed at a respective storedsubpixel offset relative to said given stored pixel location, andwherein said hardware processor is operable to calculate said lightfield component associated with said given subpixel by computationallyoffsetting a light field computation executed for said given pixel basedon said given stored pixel location in accordance with said respectivestored subpixel offset.

In one embodiment, each of said subpixels is associated with one ofdistinct colour channels, and wherein said hardware processor isoperable to distinctly compute light field components for each of saiddistinct colour channels in a compressive light field optimizationprocess.

In one embodiment, the LFSL comprises at least one of a micro lensarray, a parallax barrier, a diffractive light field barrier or acombination thereof.

In accordance with another aspect, there is provided acomputer-implemented method, automatically implemented by one or moredigital processors, to automatically adjust user perception of an inputimage to be rendered on a digital display medium via a set of pixelsthereof to at least partially address a viewer's reduced visual acuity,wherein each of the pixels comprises a combination of sub pixelsgeometrically disposed and associated therewith, wherein the digitaldisplay medium has a light field shaping layer (LFSL) disposed thereoncomprising an array of LFSL elements, the method comprising: digitallymapping the input image on an adjusted image plane designated to providethe user with a designated image perception adjustment; calculating alight field component associated with at least some of said subpixels tooutput independently adjusted subpixel data values according to saidmapping and in accordance with a stored alignment of said LFSL elementsrelative to said pixels that at least partially governs a shaping ofeach said light field component; associating said independently adjustedsubpixel data values with said subpixels; and rendering saidindependently adjusted subpixel data values thereby rendering aperceptively adjusted version of the input image when viewed through theLFSL to at least partially address the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to said digital display medium at adesignated minimum viewing distance designated such that saidperceptively adjusted version of the input image is adjusted toaccommodate the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In one embodiment, the independently adjusted subpixel data valuescorrespond to an adjusted pixel colour computed so to manifest acorresponding adjusted image colour at a corresponding location on saidadjusted image plane given said stored alignment.

In one embodiment, the independently adjusted subpixel data valuescomprise an adjusted image colour channel value.

In one embodiment, each of said subpixels is associated with a subpixellocation on said digital display medium, and wherein said calculatingcomprises calculating said light field component associated with a givensubpixel based on said subpixel location thereof.

In one embodiment, each given pixel is associated with a given storedpixel location on said digital display medium, wherein each givensubpixel for said given pixel is disposed at a respective storedsubpixel offset relative to said given stored pixel location, andwherein said calculating comprises calculating said light fieldcomponent associated with said given subpixel by computationallyoffsetting a light field computation executed for said given pixel basedon said given stored pixel location in accordance with said respectivestored subpixel offset.

In one embodiment, each of said subpixels is associated with one ofdistinct colour channels, and wherein calculating comprises distinctlycomputing light field components for each of said distinct colourchannels in a compressive light field optimization process.

In accordance with another aspect, there is provided a non-transitorycomputer-readable medium having instructions stored thereon, to beautomatically implemented by one or more digital processors, toautomatically adjust user perception of an input image to be rendered bya digital display medium via a set of pixels thereof to at leastpartially address a viewer's reduced visual acuity, wherein each of thepixels comprises a combination of subpixels geometrically disposed andassociated therewith, wherein the digital display medium has a lightfield shaping layer (LFSL) disposed thereon comprising an array of LFSLelements, the instructions comprising instructions to: digitally map theinput image on an adjusted image plane designated to provide the userwith a designated image perception adjustment; calculate a light fieldcomponent associated with at least some of said subpixels to outputindependently adjusted subpixel data values according to said map and inaccordance with a stored alignment of said LFSL elements relative tosaid pixels that at least partially governs a shaping of each said lightfield component; associate said independently adjusted subpixel datavalues with said subpixels; and render said independently adjustedsubpixel data values thereby rendering a perceptively adjusted versionof the input image when viewed through the LFSL to at least partiallyaddress the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to said digital display medium at adesignated minimum viewing distance designated such that saidperceptively adjusted version of the input image is adjusted toaccommodate the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In one embodiment, the independently adjusted subpixel data valuescorrespond to an adjusted pixel colour computed so to manifest acorresponding adjusted image colour at a corresponding location on saidadjusted image plane given said stored alignment.

In one embodiment, the independently adjusted subpixel data valuescomprise an adjusted image colour channel value.

In one embodiment, each of said subpixels is associated with a subpixellocation on said digital display medium, and wherein said instructionsto calculate comprise instructions to calculate said light fieldcomponent associated with a given subpixel based on said subpixellocation thereof.

In one embodiment, each given pixel is associated with a given storedpixel location on said digital display medium, wherein each givensubpixel for said given pixel is disposed at a respective storedsubpixel offset relative to said given stored pixel location, andwherein said instructions to calculate comprise instructions tocalculate said light field component associated with said given subpixelby computationally offsetting a light field computation executed forsaid given pixel based on said given stored pixel location in accordancewith said respective stored subpixel offset.

In one embodiment, each of said subpixels is associated with one ofdistinct colour channels, and wherein said instructions to calculatecomprise instructions to distinctly compute light field components foreach of said distinct colour channels in a compressive light fieldoptimization process.

In accordance with another aspect, there is provided a digital displaydevice to render an image for viewing by a viewer having reduced visualacuity, the device comprising: a digital display medium comprising anarray of pixels and operable to render a pixelated image accordingly; amicrolens array disposed relative to said digital display so to aligneach said microlens with a corresponding set of said pixels to shape alight field emanating therefrom and thereby at least partially govern aprojection thereof from said display medium toward the user; and ahardware processor operable on pixel data for the image to be displayedto output corrected image pixel data to be rendered as a function of adesignated characteristic of said microlens array and a selected visioncorrection parameter related to the viewer's reduced visual acuity suchthat said processed image is rendered via said microlens array to atleast partially compensate for the user's reduced visual acuity; whereina dimension of each said microlens is selected to minimize a spot sizeon the retina of the viewer produced by given pixels of saidcorresponding set of pixels.

In accordance with another aspect, there is provided a digital displaydevice to render an input image for viewing by a viewer having reducedvisual acuity, the device comprising: a digital display mediumcomprising an array of pixels and operable to render a pixelated imageaccordingly; a microlens array disposed relative to said digital displayso to dispose each of said microlens over an underlying set of saidpixels to shape a light field emanating therefrom and thereby at leastpartially govern a projection thereof from said display medium towardthe viewer; and a hardware processor operable on pixel data for theinput image to output adjusted image pixel data to be rendered via saiddigital display medium and projected through said microlens array inaccordance with a stored alignment thereof so to produce a designatedimage perception adjustment to at least partially address the viewer'sreduced visual acuity; wherein a dimension of each said microlens isselected to minimize a spot size on a retina of the viewer produced bysaid underlying set of pixels.

In one embodiment, the hardware processor is operable to: digitally mapthe input image on an adjusted image plane designated to provide theuser with said designated image perception adjustment; associate saidadjusted image pixel data with at least some of said pixel setsaccording to said mapping; and render said adjusted image pixel data viasaid pixel sets thereby rendering a perceptively adjusted version of theinput image when viewed through said LFSL.

In one embodiment, the adjusted image plane is a virtual image planevirtually positioned relative to said digital display medium at adesignated minimum viewing distance designated such that saidperceptively adjusted version of the input image is adjusted toaccommodate the viewer's reduced visual acuity.

In one embodiment, the adjusted image plane is designated as a userretinal plane, wherein said mapping is implemented by scaling the inputimage on said retinal plane as a function of an input user eye focusaberration parameter.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 3 to about 15 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 3 to about 10 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 5 to about 10 of said pixels.

In one embodiment, the dimension of each said microlens is selected tominimize said spot size on the retina of the viewer produced by each ofa set of constituent subpixels for each said corresponding set ofpixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 3 to about 10 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 5 to about 10 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 10 to about 35 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 15 to 25 of said pixels.

In one embodiment, the microlens array is disposed at a designateddistance from said digital display that is shorter than a focal lengthof said microlens so to produce a divergent light field therefrom.

In one embodiment, the designated distance is selected such that saidmicrolens focuses on a virtual image plane generated thereby.

In accordance with another aspect, there is provided a microlens arrayfor use with a display medium comprising an array of pixels and operableto render a pixelated image accordingly to be viewed by a viewer havinga reduced visual acuity, wherein the microlens array is dimensioned tobe disposed relative to the digital display medium and comprises anarray of microlenses, each one of which being disposed, when overlaidonto the digital display medium, to be centered over a corresponding setof the pixels to shape a light field emanating therefrom and thereby atleast partially govern a projection thereof from said display mediumtoward the user, wherein a dimension of each said microlens is selectedto minimize a spot size on the retina of the viewer produced by givenpixels of said corresponding set of pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 3 to about 15 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 3 to about 10 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 5 to about 10 of said pixels.

In one embodiment, the dimension of each said microlens is selected tominimize said spot size on the retina of the viewer produced by each ofa set of constituent subpixels for each said corresponding set ofpixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 3 to about 10 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 5 to about 10 of said pixels.

In one embodiment, a diameter of each said microlens is selected tocorrespond with a dimension of about 10 to about 35 of said pixels.

In one embodiment, the diameter of each said microlens is selected tocorrespond with a dimension of about 15 to 25 of said pixels.

Other aspects, features and/or advantages will become more apparent uponreading of the following non-restrictive description of specificembodiments thereof, given by way of example only with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF THE FIGURES

Several embodiments of the present disclosure will be provided, by wayof examples only, with reference to the appended drawings, wherein:

FIG. 1 is a diagrammatical view of an electronic device having a digitaldisplay, in accordance with one embodiment;

FIGS. 2A and 2B are exploded and side views, respectively, of anassembly of a light field display for an electronic device, inaccordance with one embodiment;

FIGS. 3A, 3B and 3C schematically illustrate normal vision, blurredvision, and corrected vision in accordance with one embodiment,respectively;

FIG. 4 is a schematic diagram of a single light field pixel defined by aconvex lenslet or microlens overlaying an underlying pixel array anddisposed at or near its focus to produce a substantially collimatedbeam, in accordance with one embodiment;

FIG. 5 is another schematic exploded view of an assembly of a lightfield display in which respective pixel subsets are aligned to emitlight through a corresponding microlens or lenslet, in accordance withone embodiment;

FIG. 6 is an exemplary diagram of a light field pattern that, whenproperly projected by a light field display, produces a corrected imageexhibiting reduced blurring for a viewer having reduced visual acuity,in accordance with one embodiment;

FIGS. 7A and 7B are photographs of a Snellen chart, as illustrativelyviewed by a viewer with reduced acuity without image correction (blurryimage in FIG. 7A) and with image correction via a light field display(corrected image in FIG. 7B), in accordance with one embodiment;

FIG. 8 is a schematic diagram of a portion of hexagonal lenslet arraydisposed at an angle relative to an underlying pixel array and having anirregular pitch mismatch offset, in accordance with one embodiment;

FIGS. 9A and 9B are photographs as illustratively viewed by a viewerwith reduced visual acuity without image correction (blurry image inFIG. 9A) and with image correction via a light field display having anangularly mismatched lenslet array (corrected image in FIG. 9B), inaccordance with one embodiment;

FIGS. 10A and 10B are photographs as illustratively viewed by a viewerwith reduced visual acuity without image correction (blurry image inFIG. 10A) and with image correction via a light field display having anangularly mismatched lenslet array (corrected image in FIG. 10B), inaccordance with one embodiment;

FIG. 11 is a process flow diagram of an illustrative ray-tracingrendering process, in accordance with one embodiment;

FIGS. 12 and 13 are process flow diagrams of exemplary input constantparameters and variables, respectively, for the ray-tracing renderingprocess of FIG. 11 , in accordance with one embodiment;

FIGS. 14A to 14C are schematic diagrams illustrating certain processsteps of FIG. 11 ;

FIG. 15 is a process flow diagram of an exemplary process for computingthe center position of an associated light field shaping unit in theray-tracing rendering process of FIG. 11 , in accordance with oneembodiment;

FIGS. 16A and 16B are schematic diagrams illustrating an exemplaryhexagonal light field shaping layer with a corresponding hexagonal tilearray, in accordance with one embodiment;

FIGS. 17A and 17B are schematic diagrams illustrating overlaying astaggered rectangular tile array over the hexagonal tile array of FIGS.16A and 16B, in accordance with one embodiment;

FIGS. 18A to 18C are schematic diagrams illustrating the associatedregions of neighboring hexagonal tiles within a single rectangular tile,in accordance with one embodiment;

FIG. 19 is process flow diagram of an illustrative ray-tracing renderingprocess, in accordance with another embodiment;

FIGS. 20A to 20D are schematic diagrams illustrating certain processsteps of FIG. 19 ;

FIGS. 21A and 21B are schematic diagrams illustrating subpixelrendering, in accordance with one embodiment;

FIGS. 22A and 22B are schematic diagrams of an LCD pixel array definedby respective red (R), green (G) and blue (B) subpixels, and renderingan angular image edge using standard pixel and subpixel rendering,respectively, in accordance with one embodiment;

FIG. 23 is a schematic diagram of one of the pixels of FIG. 22A, showingmeasures for independently accounting for subpixels thereof to applysubpixel rendering to the display of a corrected image through a lightfield display, in accordance with one embodiment;

FIGS. 24A to 24E are graphical plots illustrating a variation of a spotsize on a user's retina as a function of a microlens pitch for setoperating criteria, in accordance with different embodiments;

FIGS. 25A to 25E are graphical plots illustrating a variation of a spotsize on a user's retina as a function of a microlens pitch for a set ofexemplary operating criteria, in accordance with different embodiments;

FIGS. 26A to 26C are schematic diagrams of a substantially collimated,substantially converging or substantially diverging beams, respectively,produced from a convex lenslet or microlens overlaying an underlyingpixel array and the associated change in spot size on a user's retina;

FIG. 27 is a graphical plot illustrating the effect of beam divergenceon the variation of a spot size on a user's retina as a function of amicrolens pitch for set operating criteria, in accordance with differentembodiments;

FIG. 28 is an illustration of a set of vertical and horizontal black andwhite striped patterns used to evaluate the vertical and horizontalsmallest resolvable point size of a light field display, in accordancewith different embodiments; and

FIGS. 29A to 29C are graphical plots illustrating a variation of thevertical and horizontal smallest resolvable point size (in mm) of alight field display as a function user-display distance (in cm), virtualimage distance (in diopters), and viewing angle (in degrees),respectively, in accordance with different embodiments.

DETAILED DESCRIPTION

The systems and methods described herein provide, in accordance withdifferent embodiments, different examples of a vision correction systemand method, light field display and optical element array therefor, suchas, for example, a microlens array. For instance, the devices, displaysand methods described herein may allow a user's perception of an inputimage to be displayed, to be adjusted or altered using the light fielddisplay. For instance, in some examples, users who would otherwiserequire corrective eyewear such as glasses or contact lenses, or againbifocals, may consume images produced by such devices, displays andmethods in clear or improved focus without the use of such eyewear.Other light field display applications, such as 3D displays and thelike, may also benefit from the solutions described herein, and thus,should be considered to fall within the general scope and nature of thepresent disclosure.

For example, some of the herein-described embodiments described hereinprovide for digital display devices, or devices encompassing suchdisplays, for use by users having reduced visual acuity, whereby imagesultimately rendered by such devices can be dynamically processed toaccommodate the user's reduced visual acuity so that they may consumerendered images without the use of corrective eyewear, as wouldotherwise be required. As noted above, embodiments are not to be limitedas such as the notions and solutions described herein may also beapplied to other technologies in which a user's perception of an inputimage to be displayed can be altered or adjusted via the light fielddisplay.

Generally, digital displays as considered herein will comprise a set ofimage rendering pixels and a light field shaping layer disposed at apreset distance therefrom so to controllably shape or influence a lightfield emanating therefrom. For instance, each light field shaping layer(i.e. microlens array) will be defined by an array of optical elementscentered over a corresponding subset of the display's pixel array tooptically influence a light field emanating therefrom and thereby governa projection thereof from the display medium toward the user, forinstance, providing some control over how each pixel or pixel group willbe viewed by the viewer's eye(s). As will be further detailed below,arrayed optical elements may include, but are not limited to, lenslets,microlenses or other such diffractive optical elements that togetherform, for example, a lenslet array; pinholes or like apertures orwindows that together form, for example, a parallax or like barrier;concentrically patterned barriers, e.g. cut outs and/or windows, such asa to define a Fresnel zone plate or optical sieve, for example, and thattogether form a diffractive optical barrier (as described, for example,in Applicant's co-pending U.S. application Ser. No. 15/910,908, theentire contents of which are hereby incorporated herein by reference;and/or a combination thereof, such as for example, a lenslet array whoserespective lenses or lenslets are partially shadowed or barriered arounda periphery thereof so to combine the refractive properties of thelenslet with some of the advantages provided by a pinhole barrier.

In operation, the display device will also generally invoke a hardwareprocessor operable on image pixel data for an image to be displayed tooutput corrected image pixel data to be rendered as a function of astored characteristic of the light field shaping layer (e.g. layerdistance from display screen, distance between optical elements (pitch),absolute relative location of each pixel or subpixel to a correspondingoptical element, properties of the optical elements (size, diffractiveand/or refractive properties, etc.), or other such properties, and aselected vision correction or adjustment parameter related to the user'sreduced visual acuity or intended viewing experience. While light fielddisplay characteristics will generally remain static for a givenimplementation (i.e. a given shaping layer will be used and set for eachdevice irrespective of the user), image processing can, in someembodiments, be dynamically adjusted as a function of the user's visualacuity or intended application so to actively adjust a distance of thevirtual image plane induced upon rendering the corrected/adjusted imagepixel data via the static optical layer, for example, or otherwiseactively adjust image processing parameters as may be considered, forexample, when implementing a viewer-adaptive pre-filtering algorithm orlike approach (e.g. compressive light field optimization), so to atleast in part govern an image perceived by the user's eye(s) givenpixel-specific light visible thereby through the layer.

Accordingly, a given device may be adapted to compensate for differentvisual acuity levels and thus accommodate different users and/or uses.For instance, a particular device may be configured to implement and/orrender an interactive graphical user interface (GUI) that incorporates adynamic vision correction scaling function that dynamically adjusts oneor more designated vision correction parameter(s) in real-time inresponse to a designated user interaction therewith via the GUI. Forexample, a dynamic vision correction scaling function may comprise agraphically rendered scaling function controlled by a (continuous ordiscrete) user slide motion or like operation, whereby the GUI can beconfigured to capture and translate a user's given slide motionoperation to a corresponding adjustment to the designated visioncorrection parameter(s) scalable with a degree of the user's given slidemotion operation. These and other examples are described in Applicant'sco-pending U.S. patent application Ser. No. 15/246,255, the entirecontents of which are hereby incorporated herein by reference.

With reference to FIG. 1 , and in accordance with one embodiment, adigital display device, generally referred to using the numeral 100,will now be described. In this example, the device 100 is generallydepicted as a smartphone or the like, though other devices encompassinga graphical display may equally be considered, such as tablets,e-readers, watches, televisions, GPS devices, laptops, desktop computermonitors, televisions, smart televisions, handheld video game consolesand controllers, vehicular dashboard and/or entertainment displays, andthe like.

In the illustrated embodiment, the device 100 comprises a processingunit 110, a digital display 120, and internal memory 130. Display 120can be an LCD screen, a monitor, a plasma display panel, an LED or OLEDscreen, or any other type of digital display defined by a set of pixelsfor rendering a pixelated image or other like media or information.Internal memory 130 can be any form of electronic storage, including adisk drive, optical drive, read-only memory, random-access memory, orflash memory, to name a few examples. For illustrative purposes, memory130 has stored in it vision correction application 140, though variousmethods and techniques may be implemented to provide computer-readablecode and instructions for execution by the processing unit in order toprocess pixel data for an image to be rendered in producing correctedpixel data amenable to producing a corrected image accommodating theuser's reduced visual acuity (e.g. stored and executable imagecorrection application, tool, utility or engine, etc.). Other componentsof the electronic device 100 may optionally include, but are not limitedto, one or more rear and/or front-facing camera(s) 150, an accelerometer160 and/or other device positioning/orientation devices capable ofdetermining the tilt and/or orientation of electronic device 100, andthe like.

For example, the electronic device 100, or related environment (e.g.within the context of a desktop workstation, vehicularconsole/dashboard, gaming or e-learning station, multimedia displayroom, etc.) may include further hardware, firmware and/or softwarecomponents and/or modules to deliver complementary and/or cooperativefeatures, functions and/or services. For example, in some embodiment,and as will be described in greater detail below, a pupil/eye trackingsystem may be integrally or cooperatively implemented to improve orenhance corrective image rending by tracking a location of the user'seye(s)/pupil(s) (e.g. both or one, e.g. dominant, eye(s)) and adjustinglight field corrections accordingly. For instance, the device 100 mayinclude, integrated therein or interfacing therewith, one or moreeye/pupil tracking light sources, such as one or more infrared (IR) ornear-IR (NIR) light source(s) to accommodate operation in limitedambient light conditions, leverage retinal retro-reflections, invokecorneal reflection, and/or other such considerations. For instance,different IR/NIR pupil tracking techniques may employ one or more (e.g.arrayed) directed or broad illumination light sources to stimulateretinal retro-reflection and/or corneal reflection in identifying atracking a pupil location. Other techniques may employ ambient or IR/NIRlight-based machine vision and facial recognition techniques tootherwise locate and track the user's eye(s)/pupil(s). To do so, one ormore corresponding (e.g. visible, IR/NIR) cameras may be deployed tocapture eye/pupil tracking signals that can be processed, using variousimage/sensor data processing techniques, to map a 3D location of theuser's eye(s)/pupil(s). In the context of a mobile device, such as amobile phone, such eye/pupil tracking hardware/software may be integralto the device, for instance, operating in concert with integratedcomponents such as one or more front facing camera(s), onboard IR/NIRlight source(s) and the like. In other user environments, such as in avehicular environment, eye/pupil tracking hardware may be furtherdistributed within the environment, such as dash, console, ceiling,windshield, mirror or similarly-mounted camera(s), light sources, etc.

With reference to FIGS. 2A and 2B, the electronic device 100, such asthat illustrated in FIG. 1 , is further shown to include a light fieldshaping layer 200 overlaid atop a display 120 thereof and spacedtherefrom via a transparent spacer 310 or other such means as may bereadily apparent to the skilled artisan. An optional transparent screenprotector is also included atop the layer 200.

For the sake of illustration, the following embodiments will bedescribed within the context of a light field shaping layer defined, atleast in part, by a lenslet array comprising an array of microlenses(also interchangeably referred to herein as lenslets) that are eachdisposed at a distance from a corresponding subset of image renderingpixels in an underlying digital display. It will be appreciated thatwhile a light field shaping layer may be manufactured and disposed as adigital screen overlay, other integrated concepts may also beconsidered, for example, where light field shaping elements areintegrally formed or manufactured within a digital screen's integralcomponents such as a textured or masked glass plate, beam-shaping lightsources or like component. Accordingly, each lenslet will predictivelyshape light emanating from these pixel subsets to at least partiallygovern light rays being projected toward the user by the display device.As noted above, other light field shaping layers may also be consideredherein without departing from the general scope and nature of thepresent disclosure, whereby light field shaping will be understood bythe person of ordinary skill in the art to reference measures by whichlight, that would otherwise emanate indiscriminately (i.e. isotropically) from each pixel group, is deliberately controlled to definepredictable light rays that can be traced between the user and thedevice's pixels through the shaping layer.

For greater clarity, a light field is generally defined as a vectorfunction that describes the amount of light flowing in every directionthrough every point in space. In other words, anything that produces orreflects light has an associated light field. The embodiments describedherein produce light fields from an object that are not “natural” vectorfunctions one would expect to observe from that object. This gives itthe ability to emulate the “natural” light fields of objects that do notphysically exist, such as a virtual display located far behind the lightfield display, which will be referred to now as the ‘virtual image’. Asnoted in the examples below, in some embodiments, lightfield renderingmay be adjusted to effectively generate a virtual image on a virtualimage plane that is set at a designated distance from an input userpupil location, for example, so to effectively push back, or moveforward, a perceived image relative to the display device inaccommodating a user's reduced visual acuity (e.g. minimum or maximumviewing distance). In yet other embodiments, lightfield rendering mayrather or alternatively seek to map the input image on a retinal planeof the user, taking into account visual aberrations, so to adaptivelyadjust rendering of the input image on the display device to produce themapped effect. Namely, where the unadjusted input image would otherwisetypically come into focus in front of or behind the retinal plane(and/or be subject to other optical aberrations), this approach allowsto map the intended image on the retinal plane and work therefrom toaddress designated optical aberrations accordingly. Using this approach,the device may further computationally interpret and compute virtualimage distances tending toward infinity, for example, for extreme casesof presbyopia. This approach may also more readily allow, as will beappreciated by the below description, for adaptability to other visualaberrations that may not be as readily modeled using a virtual image andimage plane implementation. In both of these examples, and likeembodiments, the input image is digitally mapped to an adjusted imageplane (e.g. virtual image plane or retinal plane) designated to providethe user with a designated image perception adjustment that at leastpartially addresses designated visual aberrations. Naturally, whilevisual aberrations may be addressed using these approaches, other visualeffects may also be implemented using similar techniques.

In one example, to apply this technology to vision correction, considerfirst the normal ability of the lens in an eye, as schematicallyillustrated in FIG. 3A, where, for normal vision, the image is to theright of the eye (C) and is projected through the lens (B) to the retinaat the back of the eye (A). As comparatively shown in FIG. 3B, the poorlens shape (F) in presbyopia causes the image to be focused past theretina (D) forming a blurry image on the retina (E). The dotted linesoutline the path of a beam of light (G). Naturally, other visualaberrations can and will have different impacts on image formation onthe retina. To address these aberrations, a light field display (K), inaccordance with some embodiments, projects the correct sharp image (H)to the back of the retina for an eye with a lens which otherwise couldnot adjust sufficiently to produce a sharp image. The other two lightfield pixels (I) and (J) are drawn lightly, but would otherwise fill outthe rest of the image.

As will be appreciated by the skilled artisan, a light field as seen inFIG. 3C cannot be produced with a ‘normal’ two-dimensional displaybecause the pixels' light field emits light isotropically. Instead it isnecessary to exercise tight control on the angle and origin of the lightemitted, for example, using a microlens array or other light fieldshaping layer such as a parallax barrier, or combination thereof.

Following with the example of a microlens array, FIG. 4 schematicallyillustrates a single light field pixel defined by a convex microlens (B)disposed at its focus from a corresponding subset pixels in an LCDdisplay (C) to produce a substantially collimated beam of light emittedby these pixels, whereby the direction of the beam is controlled by thelocation of the pixel(s) relative to the microlens. The single lightfield pixel produces a beam similar to that shown in FIG. 3C where theoutside rays are lighter and the majority inside rays are darker. TheLCD display (C) emits light which hits the microlens (B) and it resultsin a beam of substantially collimated light (A).

Accordingly, upon predictably aligning a particular microlens array witha pixel array, a designated “circle” of pixels will correspond with eachmicrolens and be responsible for delivering light to the pupil throughthat lens. FIG. 5 schematically illustrates an example of a light fielddisplay assembly in which a microlens array (A) sits above an LCDdisplay on a cellphone (C) to have pixels (B) emit light through themicrolens array. A ray-tracing algorithm can thus be used to produce apattern to be displayed on the pixel array below the microlens in orderto create the desired virtual image that will effectively correct forthe viewer's reduced visual acuity. FIG. 6 provides an example of such apatter for the letter “Z”.

As will be detailed further below, the separation between the microlensarray and the pixel array as well as the pitch of the lenses can beselected as a function of various operating characteristics, such as thenormal or average operating distance of the display, and/or normal oraverage operating ambient light levels.

Further, as producing a light field with angular resolution sufficientfor accommodation correction over the full viewing ‘zone’ of a displaywould generally require an astronomical pixel density, instead, acorrect light field can be produced, in some embodiments, only at thelocation of the user's pupils. To do so, the light field display can bepaired with pupil tracking technology to track a location of the user'seyes/pupils relative to the display. The display can then compensate forthe user's eye location and produce the correct virtual image, forexample, in real time.

In some embodiments, the light field display can render dynamic imagesat over 30 frames per second on the hardware in a smartphone.

In some embodiments, the light field display can display a virtual imageat optical infinity, meaning that any level of accommodation-basedpresbyopia (e.g. first order) can be corrected for.

In some further embodiments, the light field display can both push theimage back and forward, thus allowing for selective image correctionsfor both hyperopia (farsightedness) and myopia (nearsightedness).

In order to demonstrate a working light field solution, and inaccordance with one embodiment, the following test was set up. A camerawas equipped with a simple lens, to simulate the lens in a human eye andthe aperture was set to simulate a normal pupil diameter. The lens wasfocused to 50 cm away and a phone was mounted 25 cm away. This wouldapproximate a user whose minimal seeing distance is 50 cm and isattempting to use a phone at 25 cm.

With reading glasses, +2.0 diopters would be necessary for the visioncorrection. A scaled Snellen chart was displayed on the cellphone and apicture was taken, as shown in FIG. 7A. Using the same cellphone, butwith a light field assembly in front that uses that cellphone's pixelarray, a virtual image compensating for the lens focus is displayed. Apicture was again taken, as shown in FIG. 7B, showing a clearimprovement.

FIGS. 9A and 9B provide another example or results achieved using anexemplary embodiment, in which a colour image was displayed on the LCDdisplay of a Sony™ Xperia™ XZ Premium phone (reported screen resolutionof 3840×2160 pixels with 16:9 ratio and approximately 807 pixel-per-inch(ppi) density) without image correction (FIG. 9A) and with imagecorrection through a square fused silica microlens array set at a 2degree angle relative to the screen's square pixel array and defined bymicrolenses having a 7.0 mm focus and 200 μm pitch. In this example, thecamera lens was again focused at 50 cm with the phone positioned 30 cmaway. Another microlens array was used to produce similar results, andconsisted of microlenses having a 10.0 mm focus and 150 μm pitch.

FIGS. 10A and 10B provide yet another example or results achieved usingan exemplary embodiment, in which a colour image was displayed on theLCD display of a the Sony™ Xperia™ XZ Premium phone without imagecorrection (FIG. 10A) and with image correction through a square fusedsilica microlens array set at a 2 degree angle relative to the screen'ssquare pixel array and defined by microlenses having a 10.0 mm focus and150 μm pitch. In this example, the camera lens was focused at 66 cm withthe phone positioned 40 cm away.

Accordingly, a display device as described above and further exemplifiedbelow, can be configured to render a corrected image via the light fieldshaping layer that accommodates for the user's visual acuity. Byadjusting the image correction in accordance with the user's actualpredefined, set or selected visual acuity level, different users andvisual acuity may be accommodated using a same device configuration.That is, in one example, by adjusting corrective image pixel data todynamically adjust a virtual image distance below/above the display asrendered via the light field shaping layer, different visual acuitylevels may be accommodated.

As will be appreciated by the skilled artisan, different imageprocessing techniques may be considered, such as those introduced aboveand taught by Pamplona and/or Huang, for example, which may alsoinfluence other light field parameters to achieve appropriate imagecorrection, virtual image resolution, brightness and the like.

With reference to FIG. 8 , and in accordance with one embodiment, anmicrolens array configuration will now be described, in accordance withanother embodiment, to provide light field shaping elements in acorrective light field implementation. In this embodiment, the microlensarray 800 is defined by a hexagonal array of microlenses 802 disposed soto overlay a corresponding square pixel array 804. In doing so, whileeach microlens 802 can be aligned with a designated subset of pixels toproduce light field pixels as described above, the hexagonal-to-squarearray mismatch can alleviate certain periodic optical artifacts that mayotherwise be manifested given the periodic nature of the opticalelements and principles being relied upon to produce the desired opticalimage corrections. Conversely, a square microlens array may be favouredwhen operating a digital display comprising a hexagonal pixel array.

In some embodiments, as illustrated in FIG. 8 , the microlens array 800may further or alternatively overlaid at an angle 806 relative to theunderlying pixel array, which can further or alternatively alleviateperiod optical artifacts.

In yet some further or alternative embodiments, a pitch ratio betweenthe microlens array and pixel array may be deliberately selected tofurther or alternatively alleviate periodic optical artifacts. Forexample, a perfectly matched pitch ratio (i.e. an exact integer numberof display pixels per microlens) is most likely to induce periodicoptical artifacts, whereas a pitch ratio mismatch can help reduce suchoccurrences. Accordingly, in some embodiments, the pitch ratio will beselected to define an irrational number, or at least, an irregularratio, so to minimize periodic optical artifacts. For instance, astructural periodicity can be defined so to reduce the number ofperiodic occurrences within the dimensions of the display screen athand, e.g. ideally selected so to define a structural period that isgreater than the size of the display screen being used.

While the following example is provided within the context of amicrolens array, similar structural design considerations may be appliedwithin the context of a parallax barrier, diffractive barrier orcombination thereof.

With reference to FIGS. 24A to 24E, different plots are shownillustrating the impact certain parameters may have on the performanceof a light field display. In these particular examples, the spot size ona user's retina created by a given pixel (in microns), which generallydictates how many pixels can be projected onto the user's retina at onetime, is plotted as a function of a microlens pitch (or diameter), asmeasured in pixels (i.e. the number of pixels associated with eachmicrolens). Generally, the ideal solution will be one that minimizes thespot size on the retina for given operating characteristics.

In the calculated examples, illustrated in accordance with certainexemplary embodiments, the ratio of the pupil width (taken as 1 to 3, orpreferably 1.5 to 2 times an average pupil diameter to allow for somelevel of motion without imposing too high a threshold for pupil trackingaccuracy capabilities) to the viewing/reading distance was taken be moreor less equal to the ratio of the microlens pitch (diameter) to themicrolens focal length, the later more or less dictating the distance ofthe microlens to the pixel array output surface for optimal operation.Furthermore, the viewing/reading distance may also be constrained by thetype of application. For example, for a mobile phone display, theviewing distance may be between 30 to 40 cm, while for a digital displaylocated within a vehicle (i.e. vehicular dashboard), the user may beconstrained to be positioned at a distance of at least 65 cm or similar.

Furthermore, by matching the two ratios as described above, the pitchratio (number of pixels per microlens) may then be found to be equal to:(pupil-diameter/viewing-distance)*(focal-distance/pixel-size). In someembodiments, this equation was found to give increased image quality forvalues of 8 or more. Similarly, in some embodiments, the effect of theperiodic optical artifacts mentioned above were found to be minimized,in some embodiments, with the hexagonal microlens array/square pixelarray arrangement and a pitch ratio of 8 pixels per microlens or more.

To further refine computed results, in some embodiments, a subpixelcoverage factor was considered, namely defining a percentage of thepixel area that is in fact represented by a subpixel light source, thusrefining results to actual sub pixel spot size coverage on the user'sretina. For instance, illustrated results were computed for a typical 4Kphone display in which a subpixel coverage was set at 100% (FIG. 24A),35% (FIG. 24B) and 15% (FIG. 24C). Similar results were also computedfor an 8K and 16K display (FIGS. 24D and 24E, respectively), in each ofwhich a subpixel coverage was set at 15%.

In each example, an average device viewing distance was set at 40 cm,with a minimum user focal distance set at 6 meters (to define a maximumimage correction range, i.e. the furthest a virtual image can be pushedback to accommodate reduced visual acuity in one example), as well asassuming an average eye depth of 25 mm and pupil width of 5 mm.

In some embodiments, an active percentage or fraction of each lens, i.e.a relative transparent area centered on each lens and otherwisecircumscribed by a darkened or masked periphery, such as to combine thebenefits of microlens beam shaping with parallax barrier effects, canalso be considered to further refine optimizations.

Using any of these calculations, an appropriate microlens array may beselected, for example, based on an intended application (viewerdistance), display screen technology (pixel size, subpixel coverage,pixel resolution), visual aberration correction, etc. For example, usingthe results of FIG. 24B for a 4K screen having 35% subpixel coverage,microlenses having a diameter corresponding to anywhere between about 5and 10 pixels should produce optimal results (as compared to 10-15pixels for 100% subpixel coverage and 3-7 pixels for 15% subpixelcoverage on a similarly defined 4K screen; 3-10 pixels on an 8K screen;and 5-15 for a 16K screen). As smaller microlens diameters result inbeing able to optimally bring the microlens array closer to the displayscreen, these calculations may be used to seek out a smallest diameterpossible without unduly limiting optimal retinal spot size formation.The optimized microlens diameter can also be used to select themicrolens focus length as prescribed by the above-note ratios.

While the above-described embodiments are sufficient for providing highquality light-field corrected images. In some embodiments, it may beadvantageous to further include additional parameters, such as the widthof the light beam emerging from a microlens at the corneal surface andthe width of the unlit edge of the beam as it focuses behind the retina.

With reference to FIGS. 25A to 25E, and in accordance with anotherexemplary embodiment, different plots are shown illustrating the impactcertain parameters may have on the performance of a light field display.The spot size on a user's retina created by a given pixel (in microns)is again plotted as a function of a microlens pitch (or diameter), asmeasured in pixels. However, in this exemplary embodiment, the effect ofboth the width of the light beam emerging from a microlens at thecorneal surface and the width of the unlit edge of the beam as itfocuses behind the retina were incorporated. The illustrated resultswere again computed for a typical 4K phone display in which subpixelcoverage was set at 100% (FIG. 25A), 35% (FIG. 25B) and 15% (FIG. 25C).Similar results were also computed for an 8K and 16K display (FIGS. 10Dand 10E, respectively), in each of which a subpixel coverage was set at15%. However, herein the viewing distance was set to 65 cm and the pupilwidth to 4 mm (i.e. for vehicular applications). The ideal microlenspitch is once more taken as one that minimizes the spot size on theretina for given operating characteristics.

Using any of these calculations, according to this embodiment, anappropriate microlens array may be selected, for example, based on anintended application (viewer distance), display screen technology (pixelsize, subpixel coverage, pixel resolution), visual aberrationcorrection, etc. For example, using the results of FIG. 10B for a 4Kscreen having 35% subpixel coverage, microlenses having a diametercorresponding to anywhere between about 17 to 22 pixels should produceoptimal results (as compared to 30-35 pixels for 100% subpixel coverageand 10-12 pixels for 15% subpixel coverage on a similarly defined 4Kscreen; 15-20 pixels on an 8K screen; and 20-30 for a 16K screen).

Accordingly, the methods and approaches described herein allow for thenon-trivial determination of optimal optical hardware characteristicsthat can be applied to a particular display device, user application,and/or other such characteristics.

The skilled artisan will understand, based on the parameters disclosedabove, that different combinations of microlens array/digital displaymay be available, depending on the application and applied minimum spotsize. The table below lists an exemplary set of digital displays thathave been used to demonstrate a working light field solution:

TABLE 1 Resolution Pixel size Subpixel coverage DISPLAY # (x/y pixels)(microns) (x%, y%) 1 3840 × 2160 31.5 (38, 67) 2 3840 × 2160 72.0 (27,70) 3 1920 × 1080 8.1 (50, 50) 4 2048 × 1080 47.25 (23, 60)The table includes the display's resolution, the pixel size in micronsand the subpixel coverage (in % both the x and y directions). Notably,display #1 is taken from the Sony™ Xperia™ XZ Premium phone while theother displays are attainable from manufacturers or readily modifiedfrom other devices.

Similarly, the table below lists some exemplary microlens array designsoperable to be used with some or all the displays disclosed in the tableabove:

TABLE 2 Focal Pitch Viewing MLA # Manufacturer length (mm) (mm) distance(cm) Geometry 1 Fraunhofer 43.0 0.521 65 hexagonal 2 Fraunhofer 65.01.000 65 hexagonal 3 Fraunhofer 43.0 0.600 65 hexagonal 4 Edmund Optics15.3 0.500 30 square 5 RPC 3.5 0.125 30 square 6 Okotech 10.0 0.150 30square 7 Okotech 7.0 0.200 30 square

For each microlens array listed, indicated are the focal length (mm),the pitch of each individual microlenses (in mm), the expected optimalviewing distance (in cm) and its geometry (hexagonal or square). Whileusing any combination of microlens array/display are operable to providea functional light field, some combinations may provide higher perceivedresolutions (smaller spot size on the user's retina), based on theparameters discussed above. For example, the Fraunhofer microlens arrays(#1, #2 and #3) were specifically designed to be used with display #1,based on the above-described optimization of the functional parametersaffecting the spot size on the user's retina. In contrast, the EdmundOptics microlens array (#4) is a small microlens (10 mm×10 mm) used forpreliminary testing with a focal length best suited for a 30 cm viewingdistance with the Z5 screen (display #1) while the RPC microlens array(#5) was used for a close-to-eye test, generally only a few centimetersfrom the eye. Finally, the Okotech 10 mm (#6) and 7 mm (#7) are off theshelf MLAs. Examples of such combinations of digital displays/microlensarrays from the above tables will be discussed below.

In order to demonstrate a working light field solution, and inaccordance with one embodiment, the following test was set up. A camerawas equipped with a simple lens, to simulate the lens in a human eye andthe aperture was set to simulate a normal pupil diameter. These factorsare generally controlled to adequately simulate the depth of focus ofthe human eye. The lens was focused to 50 cm away and a phone wasmounted 25 cm away. This would approximate a user whose minimal seeingdistance is 50 cm and is attempting to use a phone at 25 cm.

With reading glasses, +2.0 diopters would be necessary for the visioncorrection. A scaled Snellen chart was displayed on the cellphone and apicture was taken, as shown in FIG. 7A. Using the same cellphone, butwith a light field assembly in front that uses that cellphone's pixelarray, a virtual image compensating for the lens focus is displayed. Apicture was again taken, as shown in FIG. 7B, showing a clearimprovement.

FIGS. 9A and 9B provide another example or results achieved using anexemplary embodiment, in which a color image was displayed on the LCDdisplay #1 (a Sony™ Xperia™ XZ Premium phone with a reported screenresolution of 3840×2160 pixels with 16:9 ratio and approximately 807pixel-per-inch (ppi) density) without image correction (FIG. 9A) andwith image correction through a square fused silica microlens array setat a 2 degree angle relative to the screen's square pixel array anddefined by microlenses #7 having a 7.0 mm focus and 200 μm pitch. Inthis example, the camera lens was again focused at 50 cm with the phonepositioned 30 cm away. Another microlens array was used to producesimilar results, and consisted of microlens array #6 having microlenseswith a 10.0 mm focus and 150 μm pitch.

FIGS. 10A and 10B provide yet another example or results achieved usingan exemplary embodiment, in which a color image was displayed on the LCDdisplay of a Sony™ Xperia™ XZ Premium phone (display #1) without imagecorrection (FIG. 10A) and with image correction through a square fusedsilica microlens array set at a 2 degree angle relative to the screen'ssquare pixel array and defined by microlens array #6 having microlenseswith a 10.0 mm focus and 150 μm pitch. In this example, the camera lenswas focused at 66 cm with the phone positioned 40 cm away.

While the two above examples were provided using off-the-shelfcomponents, the Fraunhofer microlens arrays (#1, #2 and #3) describedabove were specifically designed, using the spot size minimizationprocedure described above, to be combined with display #1 (Sony™ Xperia™XZ Premium phone) and were found to provide an improved resolution thatsubstantially approaches the so-called retina resolution.

In some embodiments, the microlens array and the digital display may berelatively positioned so that each microlens produces moderatelyconvergent or divergent light beams. By adjusting the beamconvergence/divergence by changing the relative position of themicrolens and the display with respect to the microlens' focal length,the width of the spot size on the user's retina may be minimized aswell. FIGS. 26A to 26C schematically illustrates the effect using adifferent microlens focus has on the width of the spot size on theretina. Indeed, for a microlens with a pixel located on its focal plane,the emerging beam is collimated (parallel) and thus the width of thebeam on the cornea is the width of the microlens. In contrast, if thefocal plane is located instead in front/behind the pixel (e.g. pixel isfurther/closer than focal length), then the emerging beamconverges/diverges and the width of the beam on the cornea will besmaller/greater than the width (or pitch) of the microlens. In FIG. 26A,three collimated beams exiting microlens plane 2605 are shown, eachassociated with a single pixel from the display (not shown) as describedabove with respect to FIG. 4 . The rays from each beam are directedtowards the cornea/crystallin plane 2608 and converge on focal plane2615, which is located behind the retina plane 2618. The perceived spotsize 2625 is the width of the region covered by the beams on the retinaplane 2618. The associated virtual image portion covered by each beam,as perceived by the user, on the virtual image plane 2631 behindmicrolens 2605 is also shown for comparison. Similarly, FIGS. 26B and26C illustrates the perceived spot sizes 2637 and 2645, respectively,for the same configuration but with the microlens focused at the cornea(e.g. converging beam) or focused at the virtual image plane (e.g.diverging beam), respectively. For the diverging beam (FIG. 26C), theresulting spot size 2645 is smaller than the spot sizes 2625 or 2637 forthe collimated beam and the converging beam, respectively. This effectof beam divergence of the spot size is further illustrated in FIG. 27 .Starting with the same calculations used to generate FIG. 25C, a beamdivergence parameter was further added to measure the effect ofincreased beam divergence on the spot size on a user's retina created bya given pixel (in microns), herein again plotted as a function of amicrolens pitch (or diameter), as measured in pixels. For comparison,plot 2705 is the same plot as the one shown in FIG. 25C (e.g. perfectlycollimated beam) while plot 2713 is a plot obtained using the sameparameters but with an added increased beam divergence. The divergenceof the beam emerging from the microlens may be quantified using, forexample, the inverse distance from the microlens to the convergencepoint of the beam emanating from it. For a divergent beam (e.g. thatconverges “behind” the lens), this inverse distance parameter will benegative. Plot 2713 was obtained with a divergence parameter equal tothe negative inverse distance to the virtual image plane, representing amicrolens focused on the virtual image plane (same as seen in FIG. 26C).We clearly see the resulting improvement on the minimum perceived spotsize 2717 vs the original optimized value 2709, which shifts from aminimum value of about 3.3 microns to a minimum value of about 1.08microns.

With reference to FIGS. 11 to 13 , and in accordance with oneembodiment, an exemplary computationally implemented ray-tracing methodfor rendering a corrected image via the light field shaping layer (LFSL)that accommodates for the user's reduced visual acuity will now bedescribed. In this exemplary embodiment, a set of constant parameters1102 may be pre-determined. These may include, for example, any datathat are not expected to significantly change during a user's viewingsession, for instance, which are generally based on the physical andfunctional characteristics of the display for which the method is to beimplemented, as will be explained below. Similarly, every iteration ofthe rendering algorithm may use a set of input variables 1104 which areexpected to change either at each rendering iteration or at leastbetween each user's viewing session.

As illustrated in FIG. 12 , the list of constant parameters 1102 mayinclude, without limitations, the distance 1204 between the display andthe LFSL, the in-plane rotation angle 1206 between the display and LFSLframes of reference, the display resolution 1208, the size of eachindividual pixel 1210, the optical LFSL geometry 1212, the size of eachoptical element 1214 within the LFSL and optionally the subpixel layout1216 of the display. Moreover, both the display resolution 1208 and thesize of each individual pixel 1210 may be used to pre-determine both theabsolute size of the display in real units (i.e. in mm) and thethree-dimensional position of each pixel within the display. In someembodiments where the subpixel layout 1216 is available, the positionwithin the display of each subpixel may also be pre-determined. Thesethree-dimensional location/positions are usually calculated using agiven frame of reference located somewhere within the plane of thedisplay, for example a corner or the middle of the display, althoughother reference points may be chosen. Concerning the optical layergeometry 1212, different geometries may be considered, for example ahexagonal geometry such as the one shown in FIG. 8 . Finally, bycombining the distance 1204, the rotation angle 1206, and the geometry1212 with the optical element size 1214, it is possible to similarlypre-determine the three-dimensional location/position of each opticalelement center with respect to the display's same frame of reference.

FIG. 13 meanwhile illustratively lists an exemplary set of inputvariables 1104 for method 1100, which may include any input data fedinto method 1100 that may reasonably change during a user's singleviewing session, and may thus include without limitation: the image(s)to be displayed 1306 (e.g. pixel data such as on/off, colour,brightness, etc.), the three-dimensional pupil location 1308 (e.g. inembodiments implementing active eye/pupil tracking methods) and/or pupilsize 1312 and the minimum reading distance 1310 (e.g. one or moreparameters representative of the user's reduced visual acuity orcondition). In some embodiments, the eye depth 1314 may also be used.The image data 1306, for example, may be representative of one or moredigital images to be displayed with the digital pixel display. Thisimage may generally be encoded in any data format used to store digitalimages known in the art. In some embodiments, images 1306 to bedisplayed may change at a given framerate.

The pupil location 1308, in one embodiment, is the three-dimensionalcoordinates of at least one the user's pupils' center with respect to agiven reference frame, for example a point on the device or display.This pupil location 1308 may be derived from any eye/pupil trackingmethod known in the art. In some embodiments, the pupil location 1308may be determined prior to any new iteration of the rendering algorithm,or in other cases, at a lower framerate. In some embodiments, only thepupil location of a single user's eye may be determined, for example theuser's dominant eye (i.e. the one that is primarily relied upon by theuser). In some embodiments, this position, and particularly the distancepupil distance to the screen may otherwise or additionally be ratherapproximated or adjusted based on other contextual or environmentalparameters, such as an average or preset user distance to the screen(e.g. typical reading distance for a given user or group of users;stored, set or adjustable driver distance in a vehicular environment;etc.).

In the illustrated embodiment, the minimum reading distance 1310 isdefined as the minimal focus distance for reading that the user's eye(s)may be able to accommodate (i.e. able to view without discomfort). Insome embodiments, different values of the minimum reading distance 1310associated with different users may be entered, for example, as canother adaptive vision correction parameters be considered depending onthe application at hand and vision correction being addressed.

With added reference to FIGS. 14A to 14C, once parameters 1102 andvariables 1104 have been set, the method of FIG. 11 then proceeds withstep 1106, in which the minimum reading distance 1310 (and/or relatedparameters) is used to compute the position of a virtual (adjusted)image plane 1405 with respect to the device's display, followed by step1108 wherein the size of image 1306 is scaled within the image plane1405 to ensure that it correctly fills the pixel display 1401 whenviewed by the distant user. This is illustrated in FIG. 14A, which showsa diagram of the relative positioning of the user's pupil 1415, thelight field shaping layer 1403, the pixel display 1401 and the virtualimage plane 1405. In this example, the size of image 1306 in image plane1405 is increased to avoid having the image as perceived by the userappear smaller than the display's size.

An exemplary ray-tracing methodology is described in steps 1110 to 1128of FIG. 11 , at the end of which the output color of each pixel of pixeldisplay 1401 is known so as to virtually reproduce the light fieldemanating from an image 1306 positioned at the virtual image plane 1405.In FIG. 11 , these steps are illustrated in a loop over each pixel inpixel display 1401, so that each of steps 1110 to 1126 describes thecomputations done for each individual pixel. However, in someembodiments, these computations need not be executed sequentially, butrather, steps 1110 to 1128 may executed in parallel for each pixel or asubset of pixels at the same time. Indeed, as will be discussed below,this exemplary method is well suited to vectorization and implementationon highly parallel processing architectures such as GPUs.

As illustrated in FIG. 14A, in step 1110, for a given pixel 1409 inpixel display 1401, a trial vector 1413 is first generated from thepixel's position to the center position 1417 of pupil 1415. This isfollowed in step 1112 by calculating the intersection point 1411 ofvector 1413 with the LFSL 1403.

The method then finds, in step 1114, the coordinates of the center 1416of the LFSL optical element closest to intersection point 1411. Thisstep may be computationally intensive and will be discussed in moredepth below. Once the position of the center 1416 of the optical elementis known, in step 1116, a normalized unit ray vector is generated fromdrawing and normalizing a vector 1423 drawn from center position 1416 topixel 1409. This unit ray vector generally approximates the direction ofthe light field emanating from pixel 1409 through this particular lightfield element, for instance, when considering a parallax barrieraperture or lenslet array (i.e. where the path of light travelingthrough the center of a given lenslet is not deviated by this lenslet).Further computation may be required when addressing more complex lightshaping elements, as will be appreciated by the skilled artisan. Thedirection of this ray vector will be used to find the portion of image1306, and thus the associated color, represented by pixel 1409. Butfirst, in step 1118, this ray vector is projected backwards to the planeof pupil 1415, and then in step 1120, the method verifies that theprojected ray vector 1425 is still within pupil 1415 (i.e. that the usercan still “see” it). Once the intersection position, for examplelocation 1431 in FIG. 14B, of projected ray vector 1425 with the pupilplane is known, the distance between the pupil center 1417 and theintersection point 1431 may be calculated to determine if the deviationis acceptable, for example by using a pre-determined pupil size andverifying how far the projected ray vector is from the pupil center.

If this deviation is deemed to be too large (i.e. light emanating frompixel 1409 channeled through optical element 1416 is not perceived bypupil 1415), then in step 1122, the method flags pixel 1409 asunnecessary and to simply be turned off or render a black color.Otherwise, as shown in FIG. 14C, in step 1124, the ray vector isprojected once more towards virtual image plane 1405 to find theposition of the intersection point 1423 on image 1306. Then in step1126, pixel 1409 is flagged as having the color value associated withthe portion of image 1306 at intersection point 1423.

In some embodiments, method 1100 is modified so that at step 1120,instead of having a binary choice between the ray vector hitting thepupil or not, one or more smooth interpolation function (i.e. linearinterpolation, Hermite interpolation or similar) are used to quantifyhow far or how close the intersection point 1431 is to the pupil center1417 by outputting a corresponding continuous value between 1 or 0. Forexample, the assigned value is equal to 1 substantially close to pupilcenter 1417 and gradually change to 0 as the intersection point 1431substantially approaches the pupil edges or beyond. In this case, thebranch containing step 1122 is ignored and step 1220 continues to step1124. At step 1126, the pixel color value assigned to pixel 1409 ischosen to be somewhere between the full color value of the portion ofimage 1306 at intersection point 1423 or black, depending on the valueof the interpolation function used at step 1120 (1 or 0).

In yet other embodiments, pixels found to illuminate a designated areaaround the pupil may still be rendered, for example, to produce a bufferzone to accommodate small movements in pupil location, for example, oragain, to address potential inaccuracies, misalignments or to create abetter user experience.

In some embodiments, steps 1118, 1120 and 1122 may be avoidedcompletely, the method instead going directly from step 1116 to step1124. In such an exemplary embodiment, no check is made that the rayvector hits the pupil or not, but instead the method assumes that italways does.

Once the output colors of all pixels have been determined, these arefinally rendered in step 1130 by pixel display 1401 to be viewed by theuser, therefore presenting a light field corrected image. In the case ofa single static image, the method may stop here. However, new inputvariables may be entered and the image may be refreshed at any desiredfrequency, for example because the user's pupil moves as a function oftime and/or because instead of a single image a series of images aredisplayed at a given framerate.

With reference to FIGS. 19 and 20A to 20D, and in accordance with oneembodiment, another exemplary computationally implemented ray-tracingmethod for rendering an adjusted image via the light field shaping layer(LFSL) that accommodates for the user's reduced visual acuity, forexample, will now be described. In this embodiment, the adjusted imageportion associated with a given pixel/subpixel is computed (mapped) onthe retina plane instead of the virtual image plane considered in theabove example, again in order to provide the user with a designatedimage perception adjustment. Therefore, the currently discussedexemplary embodiment shares some steps with the method of FIG. 11 .Indeed, a set of constant parameters 1402 may also be pre-determined.These may include, for example, any data that are not expected tosignificantly change during a user's viewing session, for instance,which are generally based on the physical and functional characteristicsof the display for which the method is to be implemented, as will beexplained below. Similarly, every iteration of the rendering algorithmmay use a set of input variables 1404 which are expected to changeeither at each rendering iteration or at least between each user viewingsession. The list of possible variables and constants is substantiallythe same as the one disclosed in FIGS. 12 and 13 and will thus not bereplicated here.

Once parameters 1402 and variables 1404 have been set, this secondexemplary ray-tracing methodology proceeds from steps 1910 to 1936, atthe end of which the output color of each pixel of the pixel display isknown so as to virtually reproduce the light field emanating from animage perceived to be positioned at the correct or adjusted imagedistance, in one example, so to allow the user to properly focus on thisadjusted image (i.e. having a focused image projected on the user'sretina) despite a quantified visual aberration. In FIG. 19 , these stepsare illustrated in a loop over each pixel in pixel display 1401, so thateach of steps 1910 to 1934 describes the computations done for eachindividual pixel. However, in some embodiments, these computations neednot be executed sequentially, but rather, steps 1910 to 1934 may beexecuted in parallel for each pixel or a subset of pixels at the sametime. Indeed, as will be discussed below, this second exemplary methodis also well suited to vectorization and implementation on highlyparallel processing architectures such as GPUs.

Referencing once more FIG. 14A, in step 1910 (as in step 1110), for agiven pixel in pixel display 1401, a trial vector 1413 is firstgenerated from the pixel's position to pupil center 1417 of the user'spupil 1415. This is followed in step 1912 by calculating theintersection point of vector 1413 with optical layer 1403.

From there, in step 1914, the coordinates of the optical element center1416 closest to intersection point 1411 are determined. This step may becomputationally intensive and will be discussed in more depth below. Asshown in FIG. 14B, once the position of the optical element center 1416is known, in step 1916, a normalized unit ray vector is generated fromdrawing and normalizing a vector 1423 drawn from optical element center1416 to pixel 1409. This unit ray vector generally approximates thedirection of the light field emanating from pixel 1409 through thisparticular light field element, for instance, when considering aparallax barrier aperture or lenslet array (i.e. where the path of lighttraveling through the center of a given lenslet is not deviated by thislenslet). Further computation may be required when addressing morecomplex light shaping elements, as will be appreciated by the skilledartisan. In step 1918, this ray vector is projected backwards to pupil1415, and then in step 1920, the method ensures that the projected rayvector 1425 is still within pupil 1415 (i.e. that the user can still“see” it). Once the intersection position, for example location 1431 inFIG. 14B, of projected ray vector 1425 with the pupil plane is known,the distance between the pupil center 1417 and the intersection point1431 may be calculated to determine if the deviation is acceptable, forexample by using a pre-determined pupil size and verifying how far theprojected ray vector is from the pupil center.

Now referring to FIGS. 20A to 20D, steps 1921 to 1929 of method 1900will be described. Once optical element center 1416 of the relevantoptical unit has been determined, at step 1921, a vector 2004 is drawnfrom optical element center 1416 to pupil center 1417. Then, in step1923, vector 2004 is projected further behind the pupil plane onto focalplane 2006 (location where any light rays originating from optical layer1403 would be focused by the eye's lens) to locate focus point 2008. Fora user with perfect vision, focal plane 2006 would be located at thesame location as retina plane 2010, but in this example, focal plane2006 is located behind retina plane 2006, which would be expected for auser with some form of farsightedness. The position of focal plane 2006may be derived from the user's minimum reading distance 1310, forexample, by deriving therefrom the focal length of the user's eye. Othermanually input or computationally or dynamically adjustable means mayalso or alternatively considered to quantify this parameter.

The skilled artisan will note that any light ray originating fromoptical element center 1416, no matter its orientation, will also befocused onto focus point 2008, to a first approximation. Therefore, thelocation on retina plane (2012) onto which light entering the pupil atintersection point 1431 will converge may be approximated by drawing astraight line between intersection point 1431 where ray vector 1425 hitsthe pupil 1415 and focus point 2008 on focal plane 2006. Theintersection of this line with retina plane 2010 (retina image point2012) is thus the location on the user's retina corresponding to theimage portion that will be reproduced by corresponding pixel 1409 asperceived by the user. Therefore, by comparing the relative position ofretina point 2012 with the overall position of the projected image onthe retina plane 2010, the relevant adjusted image portion associatedwith pixel 1409 may be computed.

To do so, at step 1927, the corresponding projected image centerposition on retina plane 2010 is calculated. Vector 2016 is generatedoriginating from the center position of display 1401 (display centerposition 2018) and passing through pupil center 1417. Vector 2016 isprojected beyond the pupil plane onto retina plane 2010, wherein theassociated intersection point gives the location of the correspondingretina image center 2020 on retina plane 2010. The skilled technicianwill understand that step 1927 could be performed at any moment prior tostep 1929, once the relative pupil center location 1417 is known ininput variables step 1904. Once image center 2020 is known, one can thenfind the corresponding image portion of the selected pixel/subpixel atstep 1929 by calculating the x/y coordinates of retina image point 2012relative to retina image center 2020 on the retina, scaled to the x/yretina image size 2031.

This retina image size 2031 may be computed by calculating themagnification of an individual pixel on retina plane 2010, for example,which may be approximately equal to the x or y dimension of anindividual pixel multiplied by the eye depth 1314 and divided by theabsolute value of the distance to the eye (i.e. the magnification ofpixel image size from the eye lens). Similarly, for comparison purposes,the input image is also scaled by the image x/y dimensions to produce acorresponding scaled input image 2064. Both the scaled input image andscaled retina image should have a width and height between −0.5 to 0.5units, enabling a direct comparison between a point on the scaled retinaimage 2010 and the corresponding scaled input image 2064, as shown inFIG. 20D.

From there, the image portion position 2041 relative to retina imagecenter position 2043 in the scaled coordinates (scaled input image 2064)corresponds to the inverse (because the image on the retina is inverted)scaled coordinates of retina image point 2012 with respect to retinaimage center 2020. The associated color with image portion position 2041is therefrom extracted and associated with pixel 1409.

In some embodiments, method 1900 may be modified so that at step 1920,instead of having a binary choice between the ray vector hitting thepupil or not, one or more smooth interpolation function (i.e. linearinterpolation, Hermite interpolation or similar) are used to quantifyhow far or how close the intersection point 1431 is to the pupil center1417 by outputting a corresponding continuous value between 1 or 0. Forexample, the assigned value is equal to 1 substantially close to pupilcenter 1417 and gradually change to 0 as the intersection point 1431substantially approaches the pupil edges or beyond. In this case, thebranch containing step 1122 is ignored and step 1920 continues to step1124. At step 1931, the pixel color value assigned to pixel 1409 ischosen to be somewhere between the full color value of the portion ofimage 1306 at intersection point 1423 or black, depending on the valueof the interpolation function used at step 1920 (1 or 0).

In yet other embodiments, pixels found to illuminate a designated areaaround the pupil may still be rendered, for example, to produce a bufferzone to accommodate small movements in pupil location, for example, oragain, to address potential inaccuracies or misalignments.

Once the output colors of all pixels in the display have been determined(check at step 1934 is true), these are finally rendered in step 1936 bypixel display 1401 to be viewed by the user, therefore presenting alight field corrected image. In the case of a single static image, themethod may stop here. However, new input variables may be entered andthe image may be refreshed at any desired frequency, for example becausethe user's pupil moves as a function of time and/or because instead of asingle image a series of images are displayed at a given framerate.

As will be appreciated by the skilled artisan, selection of the adjustedimage plane onto which to map the input image in order to adjust a userperception of this input image allows for different ray tracingapproaches to solving a similar challenge, that is of creating anadjusted image using the light field display that can provide anadjusted user perception, such as addressing a user's reduce visualacuity. While mapping the input image to a virtual image plane set at adesignated minimum (or maximum) comfortable viewing distance can provideone solution, the alternate solution may allow accommodation ofdifferent or possibly more extreme visual aberrations. For example,where a virtual image is ideally pushed to infinity (or effectively so),computation of an infinite distance becomes problematic. However, bydesignating the adjusted image plane as the retinal plane, theillustrative process of FIG. 19 can accommodate the formation of avirtual image effectively set at infinity without invoking suchcomputational challenges. Likewise, while first order focal lengthaberrations are illustratively described with reference to FIG. 19 ,higher order or other optical anomalies may be considered within thepresent context, whereby a desired retinal image is mapped out andtraced while accounting for the user's optical aberration(s) so tocompute adjusted pixel data to be rendered in producing that image.These and other such considerations should be readily apparent to theskilled artisan.

While the computations involved in the above described ray-tracingalgorithms (steps 1110 to 1128 of FIG. 11 or steps 1920 to 1934 of FIG.19 ) may be done on general CPUs, it may be advantageous to use highlyparallel programming schemes to speed up such computations. While insome embodiments, standard parallel programming libraries such asMessage Passing Interface (MPI) or OPENMP may be used to accelerate thelight field rendering via a general-purpose CPU, the light fieldcomputations described above are especially tailored to take advantageof graphical processing units (GPU), which are specifically tailored formassively parallel computations. Indeed, modern GPU chips arecharacterized by the very large number of processing cores, and aninstruction set that is commonly optimized for graphics. In typical use,each core is dedicated to a small neighborhood of pixel values within animage, e.g., to perform processing that applies a visual effect, such asshading, fog, affine transformation, etc. GPUs are usually alsooptimized to accelerate exchange of image data between such processingcores and associated memory, such as RGB frame buffers. Furthermore,smartphones are increasingly being equipped with powerful GPUs to speedthe rendering of complex screen displays, e.g., for gaming, video, andother image-intensive applications. Several programming frameworks andlanguages tailored for programming on GPUs include, but are not limitedto, CUDA, OpenCL, OpenGL Shader Language (GLSL), High-Level ShaderLanguage (HLSL) or similar. However, using GPUs efficiently may bechallenging and thus require creative steps to leverage theircapabilities, as will be discussed below.

With reference to FIGS. 15 to 18C and in accordance with one exemplaryembodiment, an exemplary process for computing the center position of anassociated light field shaping element in the ray-tracing process ofFIG. 11 (or FIG. 19 ) will now be described. The series of steps arespecifically tailored to avoid code branching, so as to be increasinglyefficient when run on GPUs (i.e. to avoid so called “warp divergence”).Indeed, with GPUs, because all the processors must execute identicalinstructions, divergent branching can result in reduced performance.

With reference to FIG. 15 , and in accordance with one embodiment, step1114 of FIG. 11 is expanded to include steps 1515 to 1525. A similardiscussion can readily be made in respect of step 1914 of FIG. 19 , andthus need not be explicitly detailed herein. The method receives fromstep 1112 the 2D coordinates of the intersection point 1411 (illustratedin FIG. 14A) of the trial vector 1413 with optical layer 1403. Asdiscussed with respect to the exemplary embodiment of FIG. 8 , there maybe a difference in orientation between the frames of reference of theoptical layer (hexagonal array of microlenses 802 in FIG. 8 , forexample) and of the corresponding pixel display (square pixel array 804in FIG. 8 , for example). This is why, in step 1515, these inputintersection coordinates, which are initially calculated from thedisplay's frame of reference, may first be rotated to be expressed fromthe light field shaping layer's frame of reference and optionallynormalized so that each individual light shaping element has a width andheight of 1 unit. The following description will be equally applicableto any light field shaping layer having an hexagonal geometry like theexemplary embodiment of FIG. 8 . Note however that the method steps 1515to 1525 described herein may be equally applied to any kind of lightfield shaping layer sharing the same geometry (i.e. not only a microlensarray, but pinhole arrays as well, etc.). Likewise, while the followingexample is specific to an exemplary hexagonal array of LFSL elementsdefinable by a hexagonal tile array of regular hexagonal tiles, othergeometries may also benefit from some or all of the features and/oradvantages of the herein-described and illustrated embodiments. Forexample, different hexagonal LFSL element arrays, such as stretched,skewed and/or rotated arrays may be considered, as can other nestledarray geometries in which adjacent rows and/or columns of the LFSL arrayat least partially “overlap” or inter-nest. For instance, as will bedescribed further below, hexagonal arrays and like nestled arraygeometries will generally provide for a commensurately sizedrectangular/square tile of an overlaid rectangular/square array or gridto naturally encompass distinct regions as defined by two or moreadjacent underlying nestled array tiles, which can be used to advantagein the examples provided below. In yet other embodiments, the processesdiscussed herein may be applied to rectangular and/or square LFSLelement arrays. Other LFSL element array geometries may also beconsidered, as will be appreciated by the skilled artisan upon readingof the following example, without departing from the general scope andnature of the present disclosure.

For hexagonal geometries, as illustrated in FIGS. 16A and 16B, thehexagonal symmetry of the light field shaping layer 1403 may berepresented by drawing an array of hexagonal tiles 1601, each centeredon their respective light field shaping element, so that the center of ahexagonal tile element is more or less exactly the same as the centerposition of its associated light field shaping element. Thus, theoriginal problem is translated to a slightly similar one whereby one nowneeds to find the center position 1615 of the associated hexagonal tile1609 closest to the intersection point 1411, as shown in FIG. 16B.

To solve this problem, the array of hexagonal tiles 1601 may besuperimposed on or by a second array of staggered rectangular tiles1705, in such a way as to make an “inverted house” diagram within eachrectangle, as clearly illustrated in FIG. 17A, namely defining threelinearly segregated tile regions for each rectangular tile, one regionpredominantly associated with a main underlying hexagonal tile, and twoother opposed triangular regions associated with adjacent underlyinghexagonal tiles. In doing so, the nestled hexagonal tile geometry istranslated to a rectangular tile geometry having distinct linearlysegregated tile regions defined therein by the edges of underlyingadjacently disposed hexagonal tiles. Again, while regular hexagons areused to represent the generally nestled hexagonal LFSL element arraygeometry, other nestled tile geometries may be used to representdifferent nestled element geometries. Likewise, while a nestled array isshown in this example, different staggered or aligned geometries mayalso be used, in some examples, in some respects, with reducedcomplexity, as further described below.

Furthermore, while this particular example encompasses the definition oflinearly defined tile region boundaries, other boundary types may alsobe considered provided they are amenable to the definition of one ormore conditional statements, as illustrated below, that can be used tooutput a corresponding set of binary or Boolean values that distinctlyidentify a location of a given point within one or another of theseregions, for instance, without invoking, or by limiting, processingdemands common to branching or looping decisionlogics/trees/statements/etc.

Following with hexagonal example, to locate the associated hexagon tilecenter 1615 closest to the intersection point 1411, in step 1517, themethod first computes the 2D position of the bottom left corner 1707 ofthe associated (normalized) rectangular tile element 1609 containingintersection point 1411, as shown in FIG. 17B, which can be calculatedwithout using any branching statements by the following two equations(here in normalized coordinates wherein each rectangle has a height andwidth of one unit):{right arrow over (t)}=(floor(uv _(y)), 0){right arrow over (C)} _(corner)=({right arrow over (uv)}+{right arrowover (t)})−{right arrow over (t)}where {right arrow over (uv)} is the position vector of intersectionpoint 1411 in the common frame of reference of the hexagonal andstaggered rectangular tile arrays, and the floor( ) function returns thegreatest integer less than or equal to each of the xy coordinates of{right arrow over (uv)}.

Once the position of lower left corner 1707, indicated by vector {rightarrow over (C)}_(corner) 1701, of the associated rectangular element1814 containing the intersection point 1411 is known, three regions1804, 1806 and 1807 within this rectangular element 1814 may bedistinguished, as shown in FIGS. 18A to 18C. Each region is associatedwith a different hexagonal tile, as shown in FIG. 18A, namely, eachregion is delineated by the linear boundaries of adjacent underlyinghexagonal tiles to define one region predominantly associated with amain hexagonal tile, and two opposed triangular tiles defined byadjacent hexagonal tiles on either side of this main tile. As will beappreciated by the skilled artisan, different hexagonal or nestled tilegeometries will result in the delineation of different rectangular tileregion shapes, as will different boundary profiles (straight vs. curved)will result in the definition of different boundary value statements,defined further below.

Continuing with the illustrated example, in step 1519, the coordinateswithin associated rectangular tile 1814 are again rescaled, as shown onthe axis of FIG. 18B, so that the intersection point's location, withinthe associated rectangular tile, is now represented in the rescaledcoordinates by a vector {right arrow over (d)} where each of its x and ycoordinates are given by:d _(x)=2*(uv _(x) −C _(corner) _(x) )−1d _(y)=3*(uv _(y) −C _(corner) _(y) ).Thus, the possible x and y values of the position of intersection point1411 within associated rectangular tile 1814 are now contained within−1<x<1 and 0<y<3. This will make the next step easier to compute.

To efficiently find the region encompassing a given intersection pointin these rescaled coordinates, the fact that, within the rectangularelement 1814, each region is separated by a diagonal line is used. Forexample, this is illustrated in FIG. 18B, wherein the lower left region1804 is separated from the middle “inverted house” region 1806 and lowerright region 1808 by a downward diagonal line 1855, which in therescaled coordinates of FIG. 18B, follows the simple equation y=−x.Thus, all points where x<−y are located in the lower left region.Similarly, the lower right region 1808 is separated from the other tworegions by a diagonal line 1857 described by the equation y<x.Therefore, in step 1521, the associated region containing theintersection point is evaluated by using these two simple conditionalstatements. The resulting set of two Boolean values will thus bespecific to the region where the intersection point is located. Forexample, the checks (caseL=x<y, caseR=y<x) will result in the values(caseL=true, caseR=false), (caseL=false, caseR=true) and (caseL=false,caseR=false) for intersection points located in the lower left region1804, lower right region 1808 and middle region 1806, respectively. Onemay then convert these Boolean values to floating points values, whereinusually in most programming languages true/false Boolean values areconverted into 1.0/0.0 floating point values. Thus, one obtains the set(caseL, caseR) of values of (1.0, 0.0), (0.0, 1.0) or (0.0, 0.0) foreach of the described regions above.

To finally obtain the relative coordinates of the hexagonal centerassociated with the identified region, in step 1523, the set ofconverted Boolean values may be used as an input to a single floatingpoint vectorial function operable to map each set of these values to aset of xy coordinates of the associated element center. For example, inthe described embodiment and as shown in FIG. 18C, one obtains therelative position vectors of each hexagonal center {right arrow over(r)} with the vectorial function:{right arrow over (r)}=(r _(x) , r _(y))=(0.5+0.5*(caseR−caseL),⅔−(caseR−caseL))thus, the inputs of (1.0, 0.0), (0.0, 1.0) or (0.0, 0.0) map to thepositions (0.0, −⅓), (0.5, ⅔), and (1.0, −⅓), respectively, whichcorresponds to the shown hexagonal centers 1863, 1865 and 1867 shown inFIG. 18C, respectively, in the rescaled coordinates.

Now back to FIG. 15 , we may proceed with the final step 1525 totranslate the relative coordinates obtained above to absolute 3Dcoordinates with respect to the display or similar (i.e. in mm). First,the coordinates of the hexagonal tile center and the coordinates of thebottom left corner are added to get the position of the hexagonal tilecenter in the optical layer's frame of reference. As needed, the processmay then scale back the values into absolute units (i.e. mm) and rotatethe coordinates back to the original frame of reference with respect tothe display to obtain the 3D positions (in mm) of the optical layerelement's center with respect to the display's frame of reference, whichis then fed into step 1116.

The skilled artisan will note that modifications to the above-describedmethod may also be used. For example, the staggered grid shown in FIG.17A may be translated higher by a value of ⅓ (in normalized units) sothat within each rectangle the diagonals separating each region arelocated on the upper left and right corners instead. The same generalprinciples described above still applies in this case, and the skilledtechnician will understand the minimal changes to the equations givenabove will be needed to proceed in such a fashion. Furthermore, as notedabove, different LFSL element geometries can result in the delineationof different (normalized) rectangular tile regions, and thus, theformation of corresponding conditional boundary statements and resultingbinary/Boolean region-identifying and center-locating coordinatesystems/functions.

In yet other embodiments, wherein a rectangular and/or square microlensarray is used instead of a nestled (hexagonal) array, a slightlydifferent method may be used to identify the associated LFSL element(microlens) center (step 1114). Herein, the microlens array isrepresented by an array of rectangular and/or square tiles. The method,as previously described, goes through step 1515, where the x and ycoordinates are rescaled (normalized) with respect to a microlens x andy dimension (henceforth giving each rectangular and/or square tile awidth and height of 1 unit). However, at step 1517, the floor( )function is used directly on each x and y coordinates of {right arrowover (uv)} (the position vector of intersection point 1411) to find thecoordinates of the bottom left corner associated with the correspondingsquare/rectangular tile. Therefrom, the relative coordinates of the tilecenter from the bottom left corner are added directly to obtain thefinal scaled position vector:{right arrow over (r)}=(r _(x) , r _(y))=(floor(uv _(x))+0.5, floor(uv_(y))+0.5)

Once this vector is known, the method goes directly to step 1525 wherethe coordinates are scaled back into absolute units (i.e. mm) androtated back to the original frame of reference with respect to thedisplay to obtain the 3D positions (in mm) of the optical layerelement's center with respect to the display's frame of reference, whichis then fed into step 1116.

The light field rendering methods described above (from FIGS. 11 to 20D)may also be applied, in some embodiments, at a subpixel level in orderto achieve an improved light field image resolution. Indeed, a singlepixel on a color subpixelated display is typically made of several colorprimaries, typically three colored elements—ordered (on variousdisplays) either as blue, green and red (BGR) or as red, green and blue(RGB). Some displays have more than three primaries such as thecombination of red, green, blue and yellow (RGBY) or red, green, blueand white (RGBW), or even red, green, blue, yellow and cyan (RGBYC).Subpixel rendering operates by using the subpixels as approximatelyequal brightness pixels perceived by the luminance channel. This allowsthe subpixels to serve as sampled image reconstruction points as opposedto using the combined subpixels as part of a “true” pixel. For the lightfield rendering methods as described above, this means that the centerposition of a given pixel (e.g. pixel 1401 in FIG. 14 ) is replaced bythe center positions of each of its subpixel elements. Therefore, thenumber of color samples to be extracted is multiplied by the number ofsubpixels per pixel in the digital display. The methods may then followthe same steps as described above and extract the associated imageportions of each subpixel individually (sequentially or in parallel).

In FIG. 21A, an exemplary pixel 2115 is comprised of three RBG subpixels(2130 for red, 2133 for green and 2135 for blue). Other embodiments maydeviate from this color partitioning, without limitation. When renderingper pixel, as described in FIG. 11 or in FIG. 19 , the image portion2145 associated with said pixel 2115 is sampled to extract the luminancevalue of each RGB color channels 2157, which are then all rendered bythe pixel at the same time. In the case of subpixel rendering, asillustrated in FIG. 21B, the methods find the image portion 2147associated with blue subpixel 2135. Therefore, only the subpixel channelintensity value of RGB color channels 2157 corresponding to the targetsubpixel 2135 is used when rendering (herein the blue subpixel colorvalue, the other two values are discarded). In doing so, a higheradjusted image resolution may be achieved for instance, by adjustingadjusted image pixel colours on a subpixel basis, and also optionallydiscarding or reducing an impact of subpixels deemed not to intersect orto only marginally intersect with the user's pupil.

To further illustrate embodiments making use of subpixel rendering, withreference to FIGS. 22A and 22B, a (LCD) pixel array 2200 isschematically illustrated to be composed of an array of display pixels2202 each comprising red (R) 2204, green (G) 2206, and blue (B) 2208subpixels. As with the examples provided above, to produce a light fielddisplay, a light field shaping layer, such as a microlens array, is tobe aligned to overlay these pixels such that a corresponding subset ofthese pixels can be used to predictably produce respective light fieldrays to be computed and adjusted in providing a corrected image. To doso, the light field ray ultimately produced by each pixel can becalculated knowing a location of the pixel (e.g. x,y coordinate on thescreen), a location of a corresponding light field element through whichlight emanating from the pixel will travel to reach the user's eye(s),and optical characteristics of that light field element, for example.Based on those calculations, the image correction algorithm will computewhich pixels to light and how, and output subpixel lighting parameters(e.g. R, G and B values) accordingly. As noted above, to reducecomputation load, only those pixels producing rays that will interfacewith the user's eyes or pupils may be considered, for instance, using acomplementary eye tracking engine and hardware, though other embodimentsmay nonetheless process all pixels to provide greater buffer zonesand/or a better user experience.

In the example shown in FIG. 22A, an angular edge 2209 is being renderedthat crosses the surfaces of affected pixels 2210, 2212, 2214 and 2216.Using standard pixel rendering, each affected pixel is either turned onor off, which to some extent dictates a relative smoothness of theangular edge 2209.

In the example shown in FIG. 22B, subpixel rendering is insteadfavoured, whereby the red subpixel in pixel 2210, the red and greensubpixels in pixel 2214 and the red subpixel in pixel 2216 aredeliberately set to zero (0) to produce a smoother representation of theangular edge 2209 at the expense of colour trueness along that edge,which will not be perceptible to the human eye given the scale at whichthese modifications are being applied. Accordingly, image correction canbenefit from greater subpixel control while delivering sharper images.

In order to implement subpixel rendering in the context of light fieldimage correction, in some embodiments, ray tracing calculations must beexecuted in respect of each subpixel, as opposed to in respect of eachpixel as a whole, based on a location (x,y coordinates on the screen) ofeach subpixel. Beyond providing for greater rendering accuracy andsharpness, subpixel control and ray tracing computations may accommodatedifferent subpixel configurations, for example, where subpixel mixing oroverlap is invoked to increase a perceived resolution of a highresolution screen and/or where non-uniform subpixel arrangements areprovided or relied upon in different digital display technologies.

In some embodiments, however, in order to avoid or reduce a computationload increase imparted by the distinct consideration of each subpixel,some computation efficiencies may be leveraged by taking into accountthe regular subpixel distribution from pixel to pixel, or in the contextof subpixel sharing and/or overlap, for certain pixel groups, lines,columns, etc. With reference to FIG. 23 , a given pixel 2300, much asthose illustrated in FIGS. 22A and 22B, is shown to include horizontallydistributed red (R) 2304, green (G) 2306, and blue (B) 2308 subpixels.Using standard pixel rendering and ray tracing, light emanating fromthis pixel can more or less be considered to emanate from a pointlocated at the geometric center 2310 of the pixel 2300. To implementsubpixel rendering, ray tracing could otherwise be calculated intriplicate by specifically addressing the geometric location of eachsubpixel. Knowing the distribution of subpixels within each pixel,however, calculations can be simplified by maintaining pixel-centeredcomputations and applying appropriate offsets given known geometricsubpixel offsets (i.e. negative horizontal offset 2314 for the redsubpixel 2304, a zero offset for the green 2306 and a positivehorizontal offset 2318 for the blue subpixel 2308). In doing so, lightfield image correction can still benefit from subpixel processingwithout significantly increased computation load.

While this example contemplates a linear (horizontal) subpixeldistribution, other 2D distributions may also be considered withoutdeparting from the general scope and nature of the present disclosure.For example, for a given digital display screen and pixel and subpixeldistribution, different subpixel mappings can be determined to definerespective pixel subcoordinate systems that, when applied to standardpixel-centric ray tracing and image correction algorithms, can allow forsubpixel processing and increase image correction resolution andsharpness without undue processing load increases.

In some embodiments, additional efficiencies may be leveraged on the GPUby storing the image data, for example image 1306, in the GPU's texturememory. Texture memory is cached on chip and in some situations isoperable to provide higher effective bandwidth by reducing memoryrequests to off-chip DRAM. Specifically, texture caches are designed forgraphics applications where memory access patterns exhibit a great dealof spatial locality, which is the case of the steps 1110-1126 of method1100. For example, in method 1100, image 1306 may be stored inside thetexture memory of the GPU, which then greatly improves the retrievalspeed during step 1126 where the color channel associated with theportion of image 1306 at intersection point 1423 is determined.

With reference to FIGS. 28 to 29C, and in accordance with differentembodiments, different illustrations and plots are shown illustratingthe impact certain parameters may have on the performance of a lightfield display's perceived resolution (herein defined as the size of thesmallest resolvable point size or SRPS). In these examples, an image ofa vertical and horizontal black and white striped pattern (one exampleof which is shown in FIG. 28 ) was created and displayed on the lightfield display. At first, the striped patterns featured linewidths in therange of 1 pixel to 20 pixels while another had linewidths ranging from1 to 100 pixels at twice the resolution (to increase the granularity ofthe test results). A Nikon DSLR camera with 45 mm lens and manufacturedentrance pupil was used to take images (of the light field image) tobest simulate the depth of field and focus capability of the human eye.Both the light field display and the camera were placed on an activeleveling optical bench to prevent distortion from the vibrations in thebuilding. All of the following tests were done using MLA #2 of table 2above (e.g. 65 mm focal length and 1 mm pitch) with the display #1 fromtable 1 (e.g. 4K Sony™ Xperia™ XZ Premium phone display). Unlessindicated otherwise, rendering software placed the virtual image at 6 mbehind the display and the viewing distance is 65 cm, perpendicular tothe light field display. The images were analyzed qualitatively and aSRPS value was determined based on which striped pattern had discernabledark and light striped patterns without interruption to the stripes. Thecompound lens on the camera had greater depth of field that a human eyewould, thus for areas where diopter change occurs quickly (virtual imagecloser to the camera than the display) the results were biased towardslower resolution than a human eye would perceive them. The change inSRPS as a function of different parameters were then plotted in FIGS.29A to 29D, as will be explained below.

It was generally observed that the vertical SRPS is lower than thehorizontal SRPS. This asymmetry is believed to be caused by a limitationof the display's drivers that limited subpixel rendering control in thevertical direction. For example, for the vertical orientation, it wasfound that it takes roughly 5-6 pixel-width lines, meaning a SRPS ofroughly 0.25 mm for a clearly discernible pattern to be formed. Incomparison the human eye resolution is about 0.016 degrees which at 65cm gives a SRPS of around 0.2 mm. In the presently discussed example,the horizontal SRPS is better (i.e. smaller) due to better subpixelcontrol and discernable patterns may be seen for lines 3-4 pixels apart.This would give an effective SRPS of roughly 0.2 mm or approximatelyaverage human eye resolution at a 65 cm viewing distance.

Now with reference to FIG. 29A, the change in SRPS in both horizontaland vertical directions are plotted as a function of viewing distance.We see that for distances of 35 cm and above, the closer the user is tothe display the smaller the SRPS is in both directions. However, therate of increase in SRPS at larger distances is small which may beconstrued as the light field display having a relatively high tolerancefor changes in the user viewing distance.

Similarly, FIG. 29B shows a plot illustrating the change in SRPS (in mm)in the horizontal and vertical orientation as a function of the virtualimage distance. This distance is herein measured as a diopter changefrom 65 cm, in which case 0 diopters is (a flat image). The observedSRPS in both orientations is smallest within the 1-1.4 diopter range(which includes the distance of 6 m as used above or larger towardsinfinity).

FIG. 29C shows a plot illustrating the change in SRPS (in mm) in thehorizontal and vertical orientation as a function of the viewing angle.In this example, the viewing distance from the display in thez-direction is kept fixed at 65 cm at the virtual image was placed atinfinity. The reference viewing angle is the user facing the displaythus the eye plane is perpendicular or at 90 degrees. It can be seenthat the optimal field of view (FOV) is around 30 degrees from thisreference angle and that a usable FOV (not optimal but still usable) isaround 60 degrees.

While the present disclosure describes various exemplary embodiments,the disclosure is not so limited. To the contrary, the disclosure isintended to cover various modifications and equivalent arrangementsincluded within the general scope of the present disclosure.

What is claimed is:
 1. A digital display device to render an input imagefor viewing by a viewer having reduced visual acuity, the devicecomprising: an array of pixels operable to render a pixelated imageaccordingly; a microlens array disposed relative to said array of pixelsto shape a light field emanating therefrom and thereby at leastpartially govern a projection thereof from said array of pixels towardthe viewer, wherein said microlens array has a focal length and isdisposed at a distance from said pixels that is shorter than said focallength so to produce a divergent light field therefrom; and a hardwareprocessor operable on pixel data for the input image to output adjustedimage pixel data to be rendered via said array of pixels and projectedthrough said microlens array so to produce a designated image perceptionadjustment to at least partially address the viewer's reduced visualacuity.
 2. The digital display device of claim 1, wherein said hardwareprocessor is operable to: digitally map the input image on an adjustedimage plane designated to provide the viewer with the designated imageperception adjustment; associate said adjusted image pixel data with atleast some of said array of pixels according to said mapping; and rendersaid adjusted image pixel data via said array of pixels therebyrendering a perceptively adjusted version of the input image when viewedthrough said microlens array.
 3. The digital display device of claim 2,wherein said adjusted image plane is a virtual image plane virtuallypositioned relative to said pixels at a designated minimum viewingdistance designated such that said perceptively adjusted version of theinput image is adjusted to accommodate the viewer's reduced visualacuity.
 4. The digital display device of claim 2, wherein said adjustedimage plane is designated as a user retinal plane, wherein said mappingis implemented by scaling the input image on said retinal plane as afunction of an input user eye focus aberration parameter.
 5. The digitaldisplay device of claim 1, wherein said hardware processor is operableto account for a geometric misalignment between said pixel array andsaid microlens array based on a stored mapping of said microlens arrayto said array of pixels.
 6. The digital display device of claim 1,wherein said array of pixels is angled or rotated relative to saidmicrolens array.
 7. The digital display device of claim 1, wherein oneof said array of pixels and said microlens array is defined by a squarearray whereas the other is defined by a hexagonal array.
 8. The digitaldisplay device of claim 1, wherein a pitch ratio between said array ofpixels and said microlens array is designated so to define an arraymismatch pattern that repeats on a periodic distance substantially equalto, or greater than a dimension of said array of pixels.
 9. Acomputer-implemented method, automatically implemented by one or moredigital processors, to automatically adjust user perception of an inputimage to be rendered by an array of pixels via a microlens arraydisposed relative thereto to shape a light field emanating therefrom andthereby at least partially govern a projection thereof from said arrayof pixels toward a viewer to at least partially address a viewer'sreduced visual acuity, the method comprising: digitally mapping theinput image on an adjusted image plane designated to provide the userwith a designated image perception adjustment; associating adjustedimage pixel data with at least some of said pixels according to saidmapping, and further according to a distance between said microlensarray and said pixels, wherein said distance is shorter than a focallength of said microlens array so to produce a divergent light fieldtherefrom; and rendering said adjusted image pixel data via said pixelsthereby rendering a perceptively adjusted version of the input imagewhen viewed through said microlens array to at least partially addressthe viewer's reduced visual acuity.
 10. The method of claim 9, whereinsaid associating is implemented according to a stored mapping of saidmicrolens array to said array of pixels.
 11. The method of claim 10,wherein the array of pixels is angled or rotated relative to themicrolens array.
 12. The method of claims 11, wherein said adjustedimage plane is a virtual image plane virtually positioned relative tothe array of pixels at a designated minimum viewing distance designatedsuch that said perceptively adjusted version of the input image isadjusted to accommodate the viewer's reduced visual acuity.
 13. Themethod of claim 11, wherein said adjusted image plane is designated as auser retinal plane, wherein said mapping is implemented by scaling theinput image on said retinal plane as a function of an input user eyefocus aberration parameter.
 14. A non-transitory computer-readablemedium having instructions stored thereon, to be automaticallyimplemented by one or more digital processors, to automatically adjustuser perception of an input image to be rendered by an array of pixelsvia a microlens array disposed relative thereto to shape a light fieldemanating therefrom and thereby at least partially govern a projectionthereof from said array of pixels toward a viewer to at least partiallyaddress a viewer's reduced visual acuity, the instructions comprisinginstructions to: digitally map the input image on an adjusted imageplane designated to provide the user with a designated image perceptionadjustment; associate adjusted image pixel data with at least some ofsaid pixel arrays according to said mapping, and further according to adistance between said microlens array and said pixels, wherein saiddistance is shorter than a focal length of said microlens array so toproduce a divergent light field therefrom; and render said adjustedimage pixel data via said pixels thereby rendering a perceptivelyadjusted version of the input image when viewed through said microlensarray to at least partially address the viewer's reduced visual acuity.15. The non-transitory computer-readable medium of claim 14, whereinsaid associating is implemented according to a stored mapping of saidmicrolens array to said array of pixels.
 16. The non-transitorycomputer-readable medium of claim 15, wherein the array of pixels isangled or rotated relative to the microlens array.
 17. Thenon-transitory computer-readable medium of claim 14, wherein saidadjusted image plane is a virtual image plane virtually positionedrelative to the array of pixels at a designated minimum viewing distancedesignated such that said perceptively adjusted version of the inputimage is adjusted to accommodate the viewer's reduced visual acuity. 18.The non-transitory computer-readable medium of claim 14, wherein saidadjusted image plane is designated as a user retinal plane, wherein saidmapping is implemented by scaling the input image on said retinal planeas a function of an input user eye focus aberration parameter.